dataesr/scientific-paragraphs-categorization huggingface dataset:
Calculation of gene overlap among the methods To sustain the sensitivity as a metric for performance evaluation of the tested methods, we overlapped the V-(D)-J genes, mapped during the BCR reconstruction by each tool, in a pairwise manner separately for HC and LC. To do this, we counted all the HC that were reconstructed in each dataset by at least one method, and we used this number as denominator of our calculation. This was necessary because some methods were incapable of reconstructing the entire number of HC in each dataset. Afterward, we overlapped lists of separate V-(D)-J genes reconstructed by the methods in a pairwise manner and divided this number by the denominator. The same procedure was also performed for the LC (Supplementary Table  S6 ).
Final score to evaluate the methods For each experimental dataset, we aggregated the sensitivity scores of HC and LC separately. For the HC, we summed the sensitivity values across different coverages and read lengths for all the libraries and divided this value by the number of libraries tested (where a library is a dataset with a defined read length and coverage). Given that BASIC was the only method capable of reconstructing HC in libraries of 25 bp length and that BRACER could not reconstruct HC when the read length was 50 bp in most of the libraries, we divided the sum of the sensitivity values across the different libraries of each dataset by the number of libraries in which the reconstruction was successful. The same was performed for the LC. Afterward, the two mean values obtained for the HC and the LC were added up and divided by 2.
This score represents the evaluation of each tool for a given experimental dataset. For the simulated SHM dataset, we first summed up the accuracy values of the HC reconstruction across the four SHM libraries (where each library contained HC with a specific amount of SHMs) and divided this number by the total number of libraries. We then performed the same procedure for the LC. These two values were added up and divided by 2 to obtain a score for this dataset. Finally, we computed a cumulative average score by averaging the scores obtained from the four datasets and used it as a final evaluation metric of each method.
Creation of the scBCR docker image and data usage The scBCR docker image that was used to install and run the methods tested in this study was built using Debian 10. 6 docker. It embeds various tools that share dependences of different versions that can potentially be incompatible. To avoid this, different Anaconda environments were used to separate different versions of the same software. The native image has BRACER commit 131c2b9, MiXCR 3. 0. 13 and TRUST4 1. 0. 4. The conda omics environment contains BALDR commit 461d9b0. The conda vdjpuzzle environment contains VDJPuzzle 3. 0 and BASIC 1. 5. 0. Along with the files needed to run each tool, the image also contains human genome and annotation version GRCh38, human transcripts and Bowtie index version GRCh38. The IGMT annotation is also provided to run the tools  (43).
All the tools were run using standard parameters for all the datasets, if not stated differently. For VDJPuzzle, we had to modify the first line of each fastq read when running samples from the Upadhyay dataset. Each pair of reads was modified from e. g. '@SRR6471013. 1 1' to '@ SRR6471013 1/1' for read 1, and from '@SRR6471013. 1 2' to '@ SRR6471013 1/2' for read 2. In case of BALDR, we used the output from the 'Ig mapped & Ig mapped + Unmapped' model to evaluate the reconstructed HC and LC since it was reported as the best option by the authors  (27). In case of BASIC and BRACER, we took the sequences of a given chain and used Change-O (34) within the Immcantation framework v4. 0 (34) to obtain the germline genes, productivity and in-frame V-J junctions. This information was already present in the output folders of TRUST4, VD-JPuzzle, BALDR and MiXCR.
Validation of reconstructed TT-specific BCRs We selected three pairs of reconstructed HC and LC belonging to three different BCRs for experimental validation. These TT-specific (TT+) BCRs were part of clonotypes identified in our dataset containing (TT+) memory B cells. We obtained these clonotypes by running BRACER function 'bracer summarise' (Supplementary Figure  S4 ). We synthesized and cloned variable domains of the assembled HC and LC sequences to produce and test the resulting monoclonal antibodies (mAbs). First, sequences were codon-optimized via GeneArt Gene Synthesis (Life Technologies) and the HC/LC variable genes together with 5 -BamHI and 3 -XhoI restriction sites, the Kozak sequence and the respective leader sequence were ordered from Ge-neArt (Life Technologies). The constructs were then ligated into a pcDNA3.
Effect of coverage and read length on BCR assembly, productivity and sensitivity To evaluate the experimental parameters affecting the performance of the BCR reconstruction tools in different Bcell types, we generated datasets of distinct library lengths and read coverages. To comprehensively test this, we selected three datasets with different characteristics. The first dataset named 'Leiden' consisted of reads obtained from TT-specific memory B cells and memory B cells with unknown antigen specificity. It was generated as part of this study using the SMART-seq2 protocol and had read length of 100 bp. The Canzar (SPEC-seq)  (25)  and Upadhyay (SMART-seq2)  (27)  datasets were obtained by sequencing plasmablast cells, yielding read lengths of 50 and 100 bp, respectively. We downsampled all three datasets to create libraries with variable coverage levels (from 50 000 to 1 250 000).
Leiden We observed different outcomes for the different methods in terms of the percentage of cells with assembled and productive HC and LC (Figure  2A ) using libraries generated from this dataset of 72 cells. Specifically, the number of assembled and productive HC was dependent on the number of reads for all the tools and increased with the rising coverage of the libraries. Moreover, this effect was strikingly pronounced for BASIC, which was able to assemble only <40% of the HC in libraries with 100 000 reads. Furthermore, BRACER showed difficulties in reconstructing HC in a scenario of 50-bp libraries with a very pronounced effect for coverages below 500 000 reads (Supplementary Figure  S5A ). BALDR, BRACER and TRUST4 displayed a similar high performance, assembling HC in up to 90% of the single cells.
Finally, the average sensitivity for the HC remained relatively stable across the different coverage levels with an average value of 85% for BRACER, followed by BALDR with 77% and TRUST4 with 66% (Figure  2A  and Supplementary Table  S1 ). All tools demonstrated a consistently high number of assembled and productive LC (K or L) across the different coverage and read length levels, with BALDR, BRACER and TRUST4 assembling LC in >90% of single cells (Figure  2A ). Likewise, BRACER, BALDR and TRUST4 reached LC assembly sensitivities of 99%, 94% and 93%, respectively, being the highest among the tested tools (Figure  2A  and Supplementary Table  S1 ). In conclusion, BRACER, BALDR and TRUST4 were the best performing tools, assembling HC and LC with the highest sensitivity in this dataset.
Canzar 2017 In contrast to the other two datasets, we could only test the tools using 25-and 50-bp read length scenarios due to the nature of the original libraries of 113 plasmablasts in the Canzar dataset. As in the Leiden dataset, we observed varying effects of the tested parameters on the different tools (Figure  2B ). Surprisingly, BRACER failed to assemble HC in all the tested libraries, seemingly due to the shortness of reads. On the other hand, all the other tested tools successfully assembled >75% of the productive HC. Moreover, BA-SIC reached up to 95% sensitivity across different coverage levels for the HC, followed by BALDR and TRUST4 with a sensitivity of 63% and 54%, respectively (Figure  2B  and Supplementary Table  S1 ). Although MiXCR and VDJPuzzle assembled comparably high percentages of the HC, they were less sensitive than others.
Upadhyay 2018 Comparably to the Canzar dataset, the Upadhyay dataset was originally generated by sequencing 51 single plasmablasts in a paired-end mode, using 100-bp libraries instead of 50 bp. This could potentially explain some of the differences we observed (Figure  2C ). The number of assembled and productive HC across the different coverage levels was ≥85% for all tools analyzed except for MiXCR, which assembled <60% productive chains. Consistently with the Canzar dataset, BRACER was not able to assemble HC when the length of reads dropped to 50 bp and below (Supplementary Figure  S7A ). Similarly, BASIC achieved the highest average sensitivity value (90%) across the different coverage levels for the HC. It was followed by BALDR and TRUST4 with 64% and 52% of sensitivity, respectively.
Finally, VDJPuzzle and BRACER showed an in- Original libraries were generated using either SPEC-seq or SMART-seq2 technology and had 100-, 50-and 100-bp-long reads, respectively. These datasets were downsampled to different coverages as displayed in the figures to test the effect of coverage on the assembly, productivity and sensitivity of BCR heavy and light chains (LcK or LcL). Assembled are heavy and light chains without stop codons. Productive are assembled heavy and light chains with in-frame V-J junctions. Left y-axis depicts % of assembled chains over the total number of single cells in each dataset. Right y-axis corresponds to coverage. The size of the circles is proportional to the % of the productive chains. Higher intensity of the yellow color of the circles corresponds to the higher sensitivity.
Method sensitivity is reflected by the V-(D)-J gene overlap Next, we asked whether the performance of the tested methods was driven by the mere availability of a subset of HC and LC and the corresponding ground truth sequences in each dataset (see Supplementary Figure  S1A-C ) or by the capability of each method to detect the same genes and, therefore, alleles during the mapping procedure before reconstruction. To comprehensively test this, we did a pairwise comparison of separate V-(D)-J genes that were reconstructed in HC and LC by different methods. We selected libraries of 100 bp with 1. 25 million reads from the Leiden and Upadhyay datasets and 50 bp with 1. 25 million reads from the Canzar dataset for this analysis (Figure  3 ). Starting with the Leiden dataset (Figure  3A ), we observed that BRACER and BALDR had the highest overlap of annotated HC V-(D)-J genes (0.
88, 0. 72 and 0. 84). Moreover, although MiXCR and BRACER had a high overlap of the D genes, a poor overlap of the V and J genes was reflected in a general lower sensitivity of this tool. Furthermore, BALDR and BRACER showed the highest overlap of the V and J genes of LC (0. 9 and 0. 9). These results were in line with the overall better performance of these two tools on the Leiden dataset when the ground truth was used. Besides, we observed that BASIC and BALDR had a consistently good overlap of annotated HC V-(D)-J genes (0. 71, 0. 51 and 0. 76) and the best overlap of annotated LC V-J genes (0. 86 and 0. 8) when compared to the match among other tools while using the Canzar dataset (Figure  3B ). It is important to note that BRACER did not show any overlap of HC genes with any of the tools due to the inability of BRACER to reconstruct HC using 50-bp read length libraries.
Finally, BASIC and BALDR had consistently the best overlap of V-(D)-J genes of the HC (0. 86, 0. 59 and 0. 86) and LC (0. 9 and 0. 92) in the Upadhyay dataset (Figure  3C ). Again, this agreed with the best performance of these two tools on this dataset. Overall, this analysis, which did not rely on the ground truth, reflected the performance of each method, which was evaluated when using the ground truth and the sensitivity as a metric.
Effect of SHMs on BCR assembly, production and accuracy After assessing the effect of the read coverage and length on BCR assembly and production, we next investigated the consequences of different levels of SHMs in the CDRs of the variable domains of the BCRs on the performance of each tool. To address this, we first simulated datasets that consisted of 100 HC, 100 LcK and 100 LcL and contained distinct levels of SHMs (ranging from 15 to 60) in the CDRs of their variable domains. Second, we created synthetic Illumina libraries and tested the capability of each method to reconstruct HC and LC harboring various levels of SHMs. Finally, we annotated the simulated HC and LC and used them as the ground truth to compute the accuracy of each tool (see the 'Materials and Methods' section and Supplementary Figure  S2A-D ).
The first noticeable outcome of this experiment was a pronounced decrease in the percentage of assembled HC and LC (LcK and LcL) with the increasing number of introduced SHMs in the CDRs of the variable domains for MiXCR, VDJPuzzle and TRUST4 (Figure  4 ). Interestingly, we observed a stronger effect of SHMs on the assembly of LC than HC for MiXCR and BALDR compared to other tools. In contrast, BASIC, TRUST4, VDJPuzzle and BRACER did not show large differences in the assembly of HC and LC. Importantly, not all simulated HC and LC were productive in all the datasets (Supplementary Figure  S2A-D ). In fact, the number of nonproductive chains increased with the higher number of SHMs. Consequently, using the tested tools to assemble BCRs from such libraries would result in nonproductive HC and LC.
Using the proposed metric of accuracy, we found that BRACER returned the highest average values of 95% and 97% across different SHM levels for HC and LC, respectively (Figure  4  and Supplementary Table  S1 ). This was reflected by the capability of this method to correctly assemble a chain and assign its (non)productivity as in the ground truth. Other tools, such as BASIC, BALDR and TRUST4, were also stable across the different SHM levels in HC and LC but demonstrated on average lower accuracy of 66%, 77% and 83% for the HC and 92%, 87% and 79% for the LC, respectively (Figure  4  and Supplementary Table  S1 ). In conclusion, BRACER was the most accurate tool across different SHM levels in correctly assembling HC and LC for this dataset, followed by BALDR.
Validation of the specificity of assembled BCRs To validate the specificity of the BCR sequences assembled by the different algorithms, we produced three mAbs using productive BCR sequences that were assigned to three different clonotypes by BRACER using the SMART-seq2 Leiden dataset obtained in this study. Two mAbs were based on BCR sequences of single TT-specific B cells collected from patient 1 (cells B10 and D8) and a third mAb was based on a sequence isolated from patient 2 (cell G1) (Figure  5A  and Supplementary Figure  S4 ). First, the BCR sequences of these three cells assembled by all algorithms were compared to the ground truth that was obtained using ARTI-SAN PCR followed by Sanger sequencing. The assembled sequences of 1-B10 LC, 1-D8 LC and 2-G1 HC were identical among all different algorithms and the ground truth.
We decided to produce the mAbs based on the sequences assembled by BRACER because two of the three selected sequences showed clonal relationship with BCR sequences determined by this tool in the dataset of TT-specific B cells (Figure  5C ). After cloning and expression of selected mAbs, an IgG ELISA was first performed to stratify concentrations up to 1. 5 g/ml followed by a TT ELISA to test the TT binding of these antibodies. We confirmed that an unrelated mAb with a proven specificity for citrullinated antigens was negative in the TT ELISA, while a validated TT mAb was positive. All three tested mAbs showed a clear TT binding, with 1-D8 displaying the strongest signal (Figure  5D ). In conclusion, except for sequence regions in which all the tools reported a different nucleotide composition when compared to the ground truth among themselves, only MiXCR reported individual amino acid differences in the CDR2 region.
DISCUSSION In this work, we extensively benchmarked six computational methods for BCR reconstruction using four different B-cell datasets, three experimental and one simulated. We focused our attention on the evaluation of methods capable of reconstructing full-length BCRs. The primary aim of this work was to provide guidance for the method of choice for different plate-based scRNA-seq datasets and scenarios (Figure  7C ). In addition, we directed our attention to understanding the performance of each method on highly mutated BCRs that are common in autoimmune diseases  (7, 10) , cancers  (6, 11)  and in neutralizing anti-HIV antibodies  (31) (32) (33). Methods based on either 'semi de novo' (BASIC) or 'de novo' (BALDR) assembly and using the IMGT annotation during the mapping procedure showed on average the highest performance when assembling both HC and LC The ELISA was performed with a concentration of 1.
5 g/ml of the mAbs. Validated mAbs with (pos. ctrl) and without (neg. ctrl) tetanus specificity were used as controls. The optical density (OD) was measured at 415 nm 20 min after ABTS addition. The dashed line shows the upper detection limit of the TT ELISA. Figure  6. Execution time of each method using samples with different read lengths and coverages. Each method was tested using standard parameters and libraries with different read lengths and coverages. These libraries were simulated using the Leiden dataset. Each dot represents the median value (seconds) of the execution time calculated using all single cells from a particular library type. Top x-axis depicts the read length of the libraries. Bottom x-axis shows the coverage of the same libraries. Left y-axis corresponds to the runtime in seconds.
Right y-axis shows the tested tool. The circle of each measurement corresponds to the median value of the time of execution of all the cells for a particular library type. The deviation from this value represents the standard deviation. (K or L) of BCRs (Figure  7A ) in the two plasmablast datasets (Canzar and Upadhyay). Moreover, despite the Upadhyay dataset being biased toward the overrepresented k-mers, both BALDR and BASIC algorithms maintained a good overall performance as previously reported  (27) , with BASIC being independent of the type of immunoglobulin chain. Despite having a similar high performance when compared to the other methods in the Leiden dataset, BA-SIC was, however, sensitive to the decreasing coverage level when reconstructing HC, and was outperformed by the de novo assembly-based methods BALDR, BRACER and TRUST4 (Figure  7A ).
The main difference of the Leiden dataset compared to the previously published datasets was the type of cells used for library construction, memory B cells and plasmablasts, respectively. This could have had a potential impact on the results we observed as plasmablasts produce high amounts of antibodies and thus have higher amounts of BCR mRNA. However, an excess of reads that belong to the V-(D)-J genes in a sequencing library does not strictly implicate better performance of all the tools. This could be observed in Figure  7A  for the two plasmablast datasets (Canzar and Upadhyay), in the which the final score of all the tools was lower than the one obtained for the Leiden dataset of memory B cells. This could be explained by an open problem in genomics, which describes a possibility of multiple reads, belonging to a short variable genomic region (in this case, V-D-J genes), to map to multiple locations.
To support this notion, methods based on more sophisticated algorithms to reconstruct the variable portion of the HC and LC by overlapping the unmapped reads to those mapping to V-J junctions together with genes present in the constant regions of HC and LC showed to have adopted a better strategy. Therefore, BRACER, BALDR, BASIC and TRUST4 should be chosen when assembling highly mutated sequences. For each method, a weighted average value for the sensitivity was calculated by averaging single sensitivity values that were obtained for HC and LC after running the tools on the different libraries of the three datasets: Leiden, Canzar and Upadhyay. A similar procedure was performed using the accuracy for the HC and LC obtained using the simulated SHM dataset. (B) A cumulative average score for each method across the different datasets was obtained.
(C) Our recommendation tree for method selection, according to the research question and type of the dataset to be analyzed. The experimental and simulated datasets used for this benchmarking study supplemented each other. On one hand, the experimental data reflected the real-world scenario of a limited number of available (antigen-specific) B cells and underlined the challenges in the assembly of their BCRs at different cell differentiation stages. On the other hand, larger sets of sequences with specific characteristics such as SHM load with absent sequencing challenges (e. g. duplicated reads, overrepresented k-mers, etc. ) can be compared when using simulated datasets. However, in general, simulated datasets cannot completely reflect and thus replace the in vivo situation. In terms of execution time, TRUST4, MiXCR and BA-SIC can process the highest number of reads per second.
This confirms that the adoption of these tools in the research and clinical setting would be beneficial for BCR and antibody reconstruction. We would also like to emphasize that this benchmark study was explicitly done with datasets generated using plate-based scRNA-seq techniques, such as SMART-seq2 or a variant of it named SPEC-seq. TRUST4 was the only method, among the publicly available ones, capable of processing 10x BCR data  (29). However, the unfeasibility to obtain the ground truth for 10x data and the scarcity of tools to process them made us exclude this type of data from our benchmark. Nevertheless, such comparisons should be addressed in the future studies. Moreover, although BRAPeS  (46)  was initially included in our evaluation, the extremely long runtimes to reconstruct just the CDR3 region resulted in a need to subset the reads (∼5000 as suggested by the developer).
ACKNOWLEDGEMENTS We would like to acknowledge the  Sanofi Global Postdoctoral Fellowship Program  for financing the postdoc position of Tommaso Andreani. We thank the  Flow Cytometry Core Facility (FCF) of Leiden University Medical Center (LUMC) ,  The Netherlands , for the assistance with the cell sorting. We also thank  Nima Nouri  from  Sanofi Precision Immunology group , and  Franck Rapaport  and  Taylor Sorenson  from  Sanofi 's  AI & Deep Analytics, Omics Data Science and In Silico Drug Design groups , respectively, for reading the manuscript and providing feedback.
Motivation: ANOVA Simultaneous Component Analysis (ASCA) is a popular method for the analysis of multivariate data yielded by designed experiments. Meaningful associations between factors/interactions of the experimental design and measured variables in the dataset are typically identified via significance testing, with permutation tests being the standard go-to choice. However, in settings with large numbers of variables, like omics (genomics, transcriptomics, proteomics and metabolomics) experiments, the 'holistic' testing approach of ASCA (all variables considered) often overlooks statistically significant effects encoded by only a few variables (biomarkers). Results: We hereby propose Variable-selection ASCA (VASCA), a method that generalizes ASCA through variable selection, augmenting its statistical power without inflating the Type-I error risk.