dataesr/scientific-paragraphs-categorization huggingface dataset:
(2017) , it was considered sufficient in this study. It would be possible to also optimize parameters of the hand-eye transformation at the cost of adding a few more optimization variables, but this was left out of the scope of the present paper. In fact, the uncertainty of the hand-eye calibration is often partly compensated by the offset δ, leading to a good 2D projection of the tool onto the image plane. Further research will consider optimizing the hand-eye calibration parameters in the future, especially for full 3D pose estimation of the robot. To validate the proposed algorithm, we acquired two different datasets (see Figure  7 ). The first one was generated on the benchtop, by teleoperating the robot in front (and in interaction with) a plastic model of the human digestive anatomy (1,000 images, 100 s long).
This model was manually moved by an operator in all directions during the robot teleoperation in order to simulate physiological movements. Displacements obtained in the images are complex because the model features 3D shapes. The second dataset (2,000 images, 200 s long) was obtained during an in vivo experiment in a porcine model. The surgeon performs an actual colorectal Endoscopic Submucosal Dissection by teleoperating the STRAS robot. The study protocol for this experiment was approved by the Institutional Ethical Committee on Animal Experimentation  (ICOMETH No. 38. 2011. 01. 018). Animals were managed in accordance with French laws for animal use and care as well as with the European Community Council directive no. 2010/63/EU. Images feature smoke, tissue cutting, and specular reflections.
Implementation Details The proposed algorithm was implemented in Python, using tools from OpenCV  (Bradski, 2000)  and numpy  (Oliphant, 2007)  for images and arrays manipulations, respectively. Optical flow was computed using the publicly available Tensorflow GPU code from the authors of the algorithm  (Ilg et al. , 2017)  and the full pre-trained FlowNet2 model. Both video and optical flow images have a resolution of 760 × 570 pixels, and the motors encoders outputs and video images were synchronized with a time step dt = 100 ms. Particle Swarm Optimization was implemented in python using a combination of the pyswarm library  (Lee, 2014)  and pathOS for multiprocessing support  (McKerns et al. , 2011). The algorithm was setup using different sizes of the swarm N and different iterations numbers i.
Metrics and Evaluation In order to evaluate the output of the algorithm, images were manually annotated in the form of binary masks. In total, 394 images were manually annotated in the benchtop dataset, and 359 in the in vivo dataset. Using those binary masks as ground truth, masks M generated by the kinematic model after applying the optimized correction δ were then compared using the Precision, Recall, and Intersection over Union (IoU) metrics. In order to fully characterize the algorithm using the abovedefined metrics, several parameters were explored.
Sampling Strategy Since the overall optimization runs in 9 to 15 s, it is unlikely that we will be able, even with important code optimization, to make it run in real time. This is, however, not an important problem since successive images are redundant in content and therefore only a few images should be used for generating high quality labels for subsequent classifier training. In other words, we need to sample images from the dataset. We evaluated two sampling strategies. The first one, which we call S rand , is a completely random sampling strategy. The second one, which we call S sel is based on the joint angles input. Since nonlinearities in the robot kinematics are more likely to appear at direction changes (see Figure  2 ), we select images which are sufficiently far from those direction changes.
Practically, we use a window of 3/4 of the image size for the first condition, and 2 pixels of robot optical flow speed magnitude for the second condition. For fairness of evaluation, the two sampling strategies should present the same number of images. S sel being the most restrictive, it was applied to the beginning of the video (the first 2 min for the in vivo case, the first minute for the benchtop case), thus providing 33 selected images for the in vivo dataset and 36 images for the benchtop dataset. The same number of images was then extracted using random sampling on the same part of the videos.
Figures 8, 9 present examples of optimization results obtained on both datasets (more examples provided in the Supplementary Material). Figure  8  features many examples where the optimized mask M almost perfectly matches with the contours of the instrument in the image, despite challenging conditions such as a very smooth optical flow around the instrument contour, and parts of the instrument looking very similar to its immediate surroundings [especially for the in vivo case, where the blue dye used during the surgical operation (Methylene blue) was very similar to the blueish hue of the instrument tip]. Figure  9  features a few examples of incorrect results after optimization. The top row shows a situation where the instrument tip has a quite different aspect from the rest of the instrument (in this case, the grasper has a metallic aspect while the body is covered by a black sheath).
Validation of the Cost Function In order to evaluate the effectiveness of the proposed cost function independently of the convergence algorithm used, we looked at how the values of the cost function correlate with the Precision, Recall, and IoU metrics. For all the selected images in both datasets (with S rand or S sel ), we performed random modifications of δ around the nominal pose parameters q, and evaluated both the cost function and the corresponding Precision, Recall, and IoU. The results are a large sets of cost function evaluations, for various images coming from two datasets, in a variety of situations. Spearman rank order correlation was used for estimating the correlation between the cost function and the metrics, since no linear correlation could be assumed a priori. The result is a high negative correlation of the cost function values with the considered metrics, with values of -0.
Optimization Results Figure  11  presents results obtained with the proposed algorithm for the nominal case (N = 100 and i = 10), on the benchtop and the in vivo datasets. A few observations can be made from these graphs. First, the raw kinematic model appears to provide poor quality outputs. This is all the more true on the in vivo dataset, which features ample tool-tissue interactions. Second, the optimization appears to provide a net increase in all metrics values (which was expected given results from section 5. 2), while the consistency check gives an additional improvement to those values, especially in the in vivo case. In order to quantitatively confirm these observations, we used statistical tests. After assessing the normality of the data, we performed a two-factor ANOVA. Factor S was the sampling strategy (S rand or S sel ), and factor O was the optimization type (O no , O optim , and O optim+cc ).
The effect of each factor was assessed. When statistical significance was found, post-hoc testing was performed using independent t-tests, with a Holm-Bonferroni correction in order to account for multiple testing. Finally, the size of the effect was also evaluated using the ω 2 effect size measure  (Ialongo, 2016). On the in vivo dataset, the image sampling strategy S was found to have a significant influence on the results (p < 0. 001 for all metrics), with a small effect size (ω 2 ≃ 0. 08 in all cases). Actually, one can see on Figure  11  that the metrics are slightly higher when using the image selection process than when using pure random sampling even for the unoptimized cases. The optimization type O was also highly significant for all metrics, with much larger effect sizes (see Table  1 ).
Results obtained on the benchtop dataset are less contrasted. One can see on Figure  11  that the image selection procedure S sel has little effect on the metrics, which is confirmed by the ANOVA results: for the factor S, p-values are > 0. 5 for all metrics, with  a negative ω 2 effect size (note that a negative effect size does not reflect a negative effect on the results, see Okada, 2017 for details). Actually, one can note that the results without optimization (O no ) are much better than on the in vivo case. This is due to the fact that, although the robot was teleoperated, movements are overall smoother in this dataset, and tool-tissue interaction is not as intensive as in the in vivo case. These combined facts make the uncorrected kinematic model generally more correct than in the in vivo case, making the image selection procedure less critical for obtaining good results.
On the other hand, the optimization factor provided statistically significant increases for the Precision and IoU metrics, with rather large effect sizes (see Table  1 ). For those metrics, post-hoc testing showed that both O optim and O optim+cc gave better results (i. e. , higher metrics values on average), with high statistical significance (p < 0. 001) when compared to the results obtained without optimization (O no ), when used with either S sel or S rand. The effect of the consistency check was not found to be significant when comparing O optim and O optim+cc with post-hoc testing. In fact, in this dataset, ample movement of the robot arm, combined with ample and constant movement of the background, make the flow images F easier to estimate for the FlowNet algorithm. The smoother motion also makes the differential kinematics less erroneous.
, 2014). A future, related direction of work could be to run our algorithm on various in vivo datasets, in order to be able to infer the average noise distribution in the labels obtained, in combination with approaches such as  (Sukhbaatar and Fergus, 2014). Another limitation to the present study is the fact that validation was performed on a limited number of images. This is in part due to the inherent difficulty and ethical questions raised when organizing multiple experiments on animal models. The limited number of images is however not really a problem for the intended use of our algorithm. In fact, the main interesting use of the algorithm could be for online classifier training, during a given surgery. In this use case, labels would be extracted on the fly as the surgery evolves, in order to train or fine-tune a classifier for segmenting the tool.
Extracting a few, relevant and high quality labels using our approach would, in this case be required to train a surgery-specific classifier. Having only a few labeled examples usually causes the classifier to overfit on the training data, which would not be a problem in this case since it would be highly correlated to the data used for subsequent inference queries. In this context, lightweight classifiers such as the ones used in García-Peraza-Herrera et al. (  2016 ),  Rocha et al. (2019)  could be interesting to use. It should be noted that this use case requires being able to run our algorithm substantially faster than in the present study, which we believe is achievable thanks to careful code optimization. Finally, several other aspect of the algorithm may also be improved in order to produce higher quality results.
FUNDING This work was supported by a grant from  ITMO Cancer in the framework of Plan Cancer  under the project named ROBOT; by  French  state funds managed by the  ANR  within the  Investissements d'Avenir program  under the  ANR-11-LABX-0004  ( Labex CAMI ); and by  ANR  under grant  ANR-18-CE19-0012-01  ( MACROS  project).
INTRODUCTION Prediction of protein structures is one of the most challenging problems in biology  (1). This is reflected by the large number of protein sequences known today (∼109 million) in UniProt versus the number of known protein structures (about 139 thousand) in Protein Data Bank, PDB  (2). This means homology modeling is a crucial technique to obtain structural insight  (3) , and homology modeling methods keep improving significantly  (4, 5). Loops are regions with often crucial roles in protein-protein interactions, protein function, drug design and docking of small molecules  (6) (7) (8). Successful loop modeling can lead towards accurate design and engineering of proteins, large peptides, antibodies, drugs or synthetic vaccines, to name a few  (9). Despite the development of dedicated loop modeling methods, the overall accuracy of homology models tends to be considerably lower in loop regions, and loop modeling of homology models remains an open problem  (10) (11) (12) (13).
Loop modeling approaches can be divided into ab initio  (14) (15) (16) (17) (18) (19) , data-based  (20) (21) (22) (23) (24)  and the combination of both methods  (25) (26) (27). Ab initio methods explore the conformational space to find loop conformations computationally, while data-based approaches mine a database using the geometry of flanks (a few residues before and after the loop of interest), to search for possible candidates. Many of these methods achieve successful loop predictions in exact environments (i. e. missing loops in crystal structures)  (16) (17) (18) 21, 23, 24, 27). However, few methods have been applied to the prediction of loops in perturbed situations (i. e. homology models)  (16, 21, 23). The difficulty of those cases is reflected by the much lower accuracy of the resulting loop models.
Moreover, few methods are available as web servers: principally GalaxyLoopPS2 (16), LoopIng  (23) , Sphinx  (27). In addition to these methods, there are servers for ModLoop  (28) , ArchPRED  (29) , FALC-Loop  (30)  and RCD+  (18) , which have only been tested on crystal structures, and there is SuperLooper2  (20) , which is an interactive web application rather than an automated web server. Moreover, there are MODELLER  (31) , Loopy  (32) , OSCAR-loop  (33) , Rosetta-NGK  (15) , LEAP  (17)  and M-DiSGro  (34) , available only as tools that have to be installed locally. Finally, there are several web servers that are specific for the prediction of loops in antibodies  (27, (35) (36) (37) (38). Previously we have proposed DaReUS-Loop, a databased approach using remote or unrelated structures for loop modeling  (39).
The method has been validated on benchmarks of loops extracted from CASP11 and CASP12 targets and shown to improve the accuracy of loop modeling, with respect to the state-of-the-art approaches (considering both ab initio and data-based methods). In addition, significant improvements have been obtained to predict long loops with at least 15 and at most 30 residues. Importantly, DaReUS-Loop tackles the practical application W424 Nucleic Acids Research, 2019, Vol. 47, Web Server issue of loop modeling in non-ideal conditions (homology models)  (39). Here, we describe the DaReUS-Loop web server, an automated platform for modeling or remodeling loops in the context of homology models. The web server uses the same protocol as in the original publication  (39) , except that in the final minimization, MODELLER (31) (which is not free software) has been replaced by GROMACS  (40).
Description of the web server The details of the DaReUS-Loop protocol are explained in  (39). The only difference is that final minimisation with MODELLER  (31) , which is not free software, has been replaced by GROMACS  (40)  (see Supplementary data for the details). In the current study, MODELLER was still used to generate initial homology models that were subsequently re-modelled using the DaReUS-Loop server. The minimum loop length is 2, and the maximum loop length is 30 residues. In addition, for the convenience of the user, the web server accepts a protein with up to 20 loop regions defined, and models them in parallel. This has implications on the valid combinations of loop candidates, which is why the server can run in three different modes: remodeling mode, modeling mode, and advanced modeling mode.
• Advanced modeling: In this mode, the inputs are similar to modeling mode (a gapped PDB and full protein sequence). Each loop is being modelled independently, while the other loops are omitted as gaps. This mode slightly improves the loop accuracy at the cost of introducing gaps in the final models. Note that all three modes produce the same results if the input protein has only one loop to be modelled. It is possible to define a PDB code that will be excluded from the search, for the purpose of benchmarking. In this case, close homologs (those with at least 70% sequence identity) are ignored within the search dataset. For every loop region, the server returns a maximum of 10 candidate models and a confidence score. In addition, to facilitate the quality assessment of loop candidates, the server returns a table reporting the final GROMACS energy values  (40)  and another energy measure that is the KORP score  (41)  for every loop candidate.
In case of multiple loops, a general clash report is generated, showing possible clashes between candidates of different loops. The report is useful in the advanced and remodeling modes, since there is a possibility that candidates of one loop have clashes with candidates of other loops. Therefore, the clash report guides the user how to choose different combinations of candidates for each loop to avoid possible clashes. A brief description of the protocol is shown in Figure  1. The server provides a visualisation facility using the NGL Viewer  (42). User can select the modelled loops one at a time and all the final loop candidates will be shown on the structure using different colours. This options facilitates the visual inspection of final models. All DaReUS-Loop web server results presented in the manuscript are for remodeling mode and all predictions were evaluated using the flank RMSD, as defined in  (39).
Results on a concrete example One of the CASP11 targets, T0807 is selected as an illustrative example to demonstrate the performance of our web server. From the homology model, nine loop regions are identified:  16-30, 45-58, 69-75, 82-87, 113-119, 154-162, 168-178, 234-245 and 258-262. First, the server was run in remodeling mode, providing the initial homology model and a gapped sequence to re-model the loops. In addition, the gapped model along with full sequence was given to the web server selecting first "modeling mode" and then 'advanced modeling mode'. Final predictions for loop 2 (45-59) in modeling mode and the confidence report for all the loops are shown in Figure  2A ). Next we measured the backbone RMSD between the best loop candidate (out of top 10) and the native loop conformation (PDB id: 4WGH), after superimposing the flanks.
DATA AVAILABILITY The set of all models for CASP11 and CASP12 generated and analysed during the current study, as well as the results obtained from the web server in all the three modes (remodeling, modeling and advance modeling) are provided as a dataset (Harvard Dataverse, https://doi. org/10. 7910/DVN/ ZI6ROT). The results from other methods were reported in the previous manuscript  (39).
However, previously published standards have a limited scope and utility, since they  (1)  do not cover all relevant IG/TR loci,  (2)  do not report on the quality of the sequencing run or the performance of samples and primers and/or (3) are synthetic constructs that may not reflect the complexity of native genomic DNA  [9, 18, 19]. The EuroClonality-NGS Working Group was initiated to develop, standardise and validate protocols for IG/TR NGS applications, as introduced in Langerak et al. [20]  and described in the accompanying manuscripts by Brüggemann et al. [21]  and Scheijen et al. [22]. Innovatively, the EuroClonality-NGS assays include two types of QCs, both based on basic assay components, and both fully integrated in ARResT/Interrogate  [23] , the interactive bioinformatics platform developed within the Working Group: 1.
A central polytarget QC (cPT-QC) consisting of a standardised mixture of lymphoid specimens, representing a full repertoire of IG/TR genes. It serves to assess performance biases or unusual amplification shifts in a sequencing run by tracking primer usage and comparison with stored reference profiles. 2. A central in-tube quality/quantification control (cIT-QC) consisting of human B and T cell lines with welldefined IG/TR rearrangements. The cIT-QC is directly added to a sample to undergo concurrent library preparation and sequencing, acting as in-tube qualitative and quantitative standard that is subjected to the same technical downstream variables. Here we describe, evaluate and showcase these concepts and functionalities. We tested the developed protocol on a dataset of polyclonal samples, B-ALL and T-ALL diagnostic materials and follow-ups of patients with substantial treatment-induced shifts in IG/TR repertoires.
Creation of a test dataset To evaluate and showcase the aforementioned concepts and functionalities, we compiled a test dataset with: 1. Four diagnostic bone marrow B-/T-ALL samples with high leukaemic infiltration (assessed by routine cytomorphology to be 60-80%). 2. Four samples of patients with B/T cell aplasia after antibody treatment. The two samples with B cell aplasia were CLL samples after Rituximab (anti-CD20) treatment and the two samples with T cell aplasia were T cell prolymphocytic leukaemia samples after Alemtuzumab (anti-CD52) treatment. In all these samples lineage-specific aplasia was confirmed by flow cytometry. 3. cPT-QC for all primer sets, but with the TRB-VJ primer set results swapped with perturbed results from experiments outlined above. To showcase generic QC functionalities, one diagnostic sample was subsampled to <1000 random reads.
The diagnostic samples and the cPT-QC were run with all primer sets as described in the accompanying manuscript by Brüggemann et al. [21] , while the aplastic follow-up samples only with the corresponding primer sets, that is, the IG sets for samples with B cell aplasia, and the TR sets for samples with T cell aplasia. Figure  1  includes a schematic of the test dataset. Finally, the follow-up samples were run without the addition of MNC to test that the addition of cIT-QC is sufficient to stabilise the samples for sequencing without compromising their immunogenetic profile.
Marker identification and quantification Abundances of lymphocyte subpopulations are frequently not available for samples of patients with lymphoid malignancies. Furthermore, as IG/TR NGS only reflects relative representation of the rearrangements, it was important to establish a calibrator that would allow us to normalise sequencing reads to input DNA cells. Analysis of our test dataset showed the utility of the cIT-QC in marker identification and quantification. Excluding cIT-QC reads, both diagnostic and aplastic samples seem to harbour few highly abundant clones if simply based on the number of reads (Fig. 3 , Supplementary Table  S5 ). However, the very high number of reads from only a very limited number of cIT-QC cells (120-440, dependent on the number of cIT-QC rearrangements per primer set), in all aplastic and a few of the diagnostic samples, are an indirect yet clear indication of the restricted numbers of patient cells harbouring rearrangements in those samples.
However, there is to date no other scientific or commercial solution available that exceeds our methodology in its broad applicability (universal IG/TR approach) and/or allows precise absolute quantification  [12, [27] [28] [29]. Finally, the QC protocols are embedded in ARResT/ Interrogate, which informs users with reports and messages and allows them, for example, to include the QC-failed samples back into the analysis. The logic behind this is that the 'fail' flag simply indicates that our pre-defined QC criteria were not met, and not that the data are corrupt beyond use. Nevertheless, flagged data should always be used with caution, and dependent on the application or question. In summary, our study showcases the applicability of two reference standards, developed by the EuroClonality-NGS Working Group, which allow standardised analysis of IG/TR NGS data (using the EuroClonality-NGS primer sets) with high reproducibility, accuracy and precision in marker identification.