dataesr/scientific-paragraphs-categorization huggingface dataset:
Summary of challenge/problem formulation Our goal was to dramatically accelerate association analysis for GWAS. We first collected use cases from genetic analysts to better understand the use of GWAS in our organization. The leading use case was the association of binary phenotypes with variants, using the logistic regression option in PLINK. Code profiling in representative test datasets with covariates showed that PLINK 1. 07's core logistic regression code (the fitLM() function) accounted for about 80 % of run time, and data initialization and related overhead accounted for most of the re-maining 20 %. Thus, we decided to focus first on acceleration of the regression calculation. Given this breakdown of computing time, we anticipated that an infinitely fast logistic regression routine within the context of PLINK 1.
Extracting this function out of the ∼98 000 lines of PLINK source code enabled contestants to rapidly understand and run the reference solution. Third, test data sets of the appropriate size were provided, as described in the Methods section above. A scoring mechanism was devised to reward computational efficiency and accuracy. Contestants were asked to increase performance while generating association test statistics that were identical (within 0. 1 %) to PLINK. The contestants were notified that it was acceptable to precisely compute association statistics for only the most significantly associated variant-phenotype pairs, if runtime was limiting, but did not provide any additional direction. All scores were displayed on a public real-time leaderboard. To prevent over-fitting, final scoring was calculated on 100 submission data sets that were not available to the contestants.
A 10-day contest was hosted on TopCoder. com, an online programming competition website with an existing community of over 600 000 software developers that routinely compete to solve programming challenges  [24]. The challenge attracted 320 participants, of whom 56 different contestants submitted 292 different versions of code. A prize pool of $10 000 was awarded to the top 5 contestants. It is estimated that 1120 person-hours were dedicated to this contest, making this a very cost-effective method. The five highest-scoring contest solutions were compiled on our HPC environment and benchmarked against PLINK's core logistic regression code (i. e. , the computeAssociations() reference solution), based on the average of five program runs. Strikingly, the five contest winners successfully accelerated logistic regression by 18-to 45-fold over the core logistic regression method from PLINK 1.
Numerical and computational In this category, contestants modified elements of the logistic regression calculation to increase speed. One change was to replace the standard C exp() function with a faster variant that took advantage of single-instruction multiple data (SIMD) parallelism (see below). Another was to change the numerical method used to compute matrix decompositions in the Newton-Raphson iterations. Contestants replaced the singular-value decomposition method used in the PLINK reference solution with Cholesky or QR methods  [25]. In addition, opportunistic spot modifications were made to the code in at least one case. For example, in one matrix multiplication, a contestant reordered operations to change a [matrix] * [matrix] * [vector] operation into a [matrix] * [vector] * [matrix] operation, thus saving operations.
Reintegration into PLINK To create a code product that would be as portable as PLINK and could be directly donated back to the PLINK community, we contracted with the top-scoring contestant from the logistic regression contest to have him incorporate his accelerated logistic regression method tightly into PLINK 1. 07, by replacing PLINK's fitLM() method with a drop-in replacement method that incorporated the faster code. One major advantage of our crowdsourcing efforts was to identify an expert with the skill set and the ability to solve our difficult problem. Given the winning contestants' established ability and familiarity with the code, the effort required to integrate the code was significantly less than the effort that would have been required by a third party. This modified PLINK ran logistic association analyses 3.
8-fold faster than PLINK 1. 07 in the HPC environment, approaching our initial estimate of an upper bound of a 5-fold speedup that could be achieved by accelerating the logistic regression component of the overall computational work. In the AWS environment, speedup was 7-fold (Fig. 2 ), which we attributed to a different profile of data I/O versus computation cost in that environment, compared to HPC. This modified PLINK, which we termed PLINK-FLR (fast logistic regression) was just as portable as PLINK 1. 07 and thus well suited to donation back to the PLINK community. We provided PLINK-FLR to the PLINK2 project  [26] , and the logistic regression code was adopted in PLINK 1. 9.
Additional code acceleration In addition to donating this portable code back to the PLINK community, we anticipated that we could achieve substantial additional speedups, albeit possibly less portable, by further contest-based crowdsourcing. To explore this, we further developed the code generated by the logistic regression contest. As mentioned above, for simplicity the code in the logistic regression contest took as input integer allele dosages in a text format. For real-world applications a more compact format such as the PLINK. bed/. bim/. fam fileset is required. Therefore, we extracted from PLINK the code required to read. bed/. bim/. fam filesets, added methods to make the PLINK input compatible and integrated it with contestant codes, so the contestant codes could take as input native PLINK binary filesets.
We also adjusted the output of the contestant code so that P values were generated (instead of the chi-squared statistics generated in the logistic regression contest). The end-to-end run time of the resulting program, called C1, on a test case with dimensions N = 6678, M = 645 863, P = 1, and C = 7 was 9 times faster than the PLINK 1. 07 in the AWS environment (Fig. 2 ). This speedup was accounted for by a combination of a logistic regression algorithm that was ∼35fold faster than PLINK1. 07, plus a relative reduction in data preand postprocessing time, compared to PLINK 1. 07. Importantly, part of the preprocessing time reduction was due to a change in the handling of missing genotypes. The code effectively presumes all genotypes are observed, and subjects with missing genotypes are not flagged and selectively excluded from regressions, as they are in PLINK1.
Speedup of data initialization Code C1 still included a costly preprocessing step to transform the genotype matrix from the structures used within PLINK I/O code to the structures compatible with contestant code, so unsurprisingly, data reading and initialization from PLINK. bed files emerged as a new rate-limiting step in the overall computation. We turned again to the open community to identify solutions to decrease the time required for this data initialization. The contestants were provided with code C1 and asked to revise the code however they saw fit, but were directed to the rate-limiting steps that included reading PLINK binary filesets and setup of initial genotype data structures prior to logistic regression. A winner-take-all strategy was employed for this contest. This contest awarded the first contestant to produce a solution that reduced the run-time by at least 2-fold.
This type of competition attracts fewer, but possibly more highly qualified, participants since the question is specific and the prize pool is reduced in this scenario. The winning solution accelerated the data initialization by modifying 'for' loop structures and vector initializations, removing some C++ vector operations, and eliminating an expensive transpose of the genotype matrix read from the. bed file. The end-to-end run time of the winning code (denoted C2) on a test dataset with N = 6678, M = 645 863, P = 1, and C = 7 was decreased 13-fold compared to code C1 (95 seconds vs. 1255 seconds) (Fig. 2 ).
Multithreading By design, GWAS analysis repeats the same type of calculation many times. Since modern operating systems and processors support multiple concurrent threads, GWAS analysis can take advantage of shared-memory parallel processing. We ran a contest on the TopCoder. com community to implement multithreading of our algorithm, awarding a prize to the first contestant who could achieve a 2-fold speedup of the baseline code from the second contest above. Code C2 was provided to contestants along with three sample inputs and outputs to allow contestants to test locally if their modifications functioned and gave the correct result. A 16-day contest was run to identify a solution. The winning code entry used OpenMP  [27]  to parallelize two components of the code. First, in the initialization of the genotype marker data matrix, prior to the logistic regression, a 'parallel for' construct was added to split work among threads.
Second, the core logistic regression calculation for all markers was multi-threaded, to split the outer loop over the M markers among threads. The end-to-end run time of the winning code (denoted C3) running 4 threads on a test dataset with N = 6678, M = 645 863, P = 1, and C = 7 was decreased 3. 4-fold compared to code C2 (28 seconds vs. 95 seconds) (Fig. 2 ). This was consistent with our expectation of a relative speedup that approached the number of parallel threads.
Coarse-grained parallelization and PLINK-compatible output using HPC Many investigators that run GWAS analysis, including our group, have access to high performance compute environments that use job management tools like IBM Platform LSF  [28]  or TORQUE  [29]  to do coarse-grained parallelization of calculations across many compute nodes. To help us process ever-larger genotype datasets, we wished to enable this type of scatter-gather parallelism. In addition, given that PLINK is a widely used community standard for GWAS analysis, we saw a substantial usability benefit in generating summary statistics in a format identical to PLINK 1. 07. Our crowdsourced code did not provide that format as is. Instead, it returned logistic regression P values, without the additional summary statistics such as regression coefficients and confidence intervals that are provided by PLINK.
We wished to generate PLINK-identical output reports, while avoiding the complexity of interfacing and co-compiling the contest-generated code into the PLINK 1. 07 codebase. To these ends, we established a "two-pass" analysis application. The crowd-sourced code from the multithreading contest described above (C3) was harnessed inside a script wrapper to submit parallel logistic regression jobs to the LSF scheduler. In the first analysis pass, C3 was run in parallel and using the output P values, markers were filtered according to a user-defined cutoff to exclude markers that had no significant association with the target phenotype (typically, this represents the vast majority of markers). In the second pass, the first round passing markers were submitted to PLINK-FLR, yielding standard PLINK logistic regression output files for that subset of markers that met the user's selected P value cutoff value.
Hence, the final statistical analysis output for the passing markers is in the standard PLINK 1. 07 format. Since only a very small fraction of markers have significant associations in most GWAS, this 'two pass' approach did not impose a notable performance penalty. This hybrid pipeline combining the crowd sourced code with PLINK was termed 'mPLINK. ' By distributing work across our HPC cluster, we expected to be able to rapidly process much larger datasets than possible in the single-server AWS test environment. Hence, we tested the run time of mPLINK on three datasets with sizes ranging from 4 billion to 49 billion regressions using increasing numbers of processes ranging from 1 to 50. These jobs were executed on a shared cluster that contributed some variation to run times, but the main trends were clear (Table  2 ).
Compared to PLINK1. 07, we Max speedup (fold) 591× 1177× 1450× * NA(RAM) signifies that the dataset was too large to load into memory and therefore was not calculated N refers to the number of subjects; M is the number of genetic markers (variants). observed a dataset-size dependent speed increase ranging from 591-to 1450-fold in the HPC environment. At the smallest problem size (N = 6678, M = 645 863), we observed sublinear speedups as we went from 1 to 10 processes. At this problem size, with >10 processes, the overhead of scattering and gathering coarsegrained jobs dominated, limiting speedup to no more than 591×. For the two larger datasets, where a greater fraction of the time was spent in logistic regression routines, the relative speedup was larger (up to 1450×), although still sublinear as the number of parallel processes was increased from 1 to 50, consistent with the presence of scatter-gather overheads.
We attributed the observed overall speedup to the combination of the core logistic regression speedup (developed in C1), the data initialization changes (in C2), multithreading (in C3), and the application of coarse grained parallelization. In addition to the speedup we observed, by breaking up datasets across multiple large-memory compute nodes, HPC enabled us to run datasets including a size (N = 7000, M = 7 000 000) that would have exceeded the mem-ory capacity of any widely available single-server environment. Note that we could have employed preprocessing methods like chromosome splitting to divide large datasets into smaller sizes that would not require access to large-memory environments. In our approach we used random-access into PLINK binary files to divide input datasets into flexibly-sized chunks for HPC processing, which was more convenient for us.
Real-world application We applied mPLINK to one phenotype from the COPDGene consortium dataset  [17]  (N = 6678, M = 645 863). Fig. 3B  shows Manhattan plots from this dataset generated by PLINK 1. 07 and mPLINK with a user-defined P value reporting cutoff of P ≤ 10 -3. All of the significant markers identified by PLINK 1. 07 on  Hill et al. this dataset were also identified by mPLINK. A small number of markers that were close to the user-defined cutoff were not returned by mPLINK, attributable primarily to the presence of missing values in this real-world dataset and differences in convergence criteria between calculations. The accelerated calculations in mPLINK provided us the ability to analyze and gain insight into more phenotypes in the COPDGene dataset. mPLINK was applied in the COPDGene study to analyze P = 164 binary phenotypes at a rate of <1 min/genotype, reducing analysis time from >20 days (estimated) to several hours and allowing an exhaustive survey of all binary phenotypes in the dataset.
Before the contest Before beginning the contests, the key steps we took were requirements gathering, profiling of our "current state" solution, decomposition of the problem, creation of test sets, definition of our contest scoring method, and decision on contest type. During requirements gathering we interviewed GWAS practitioners in our institution to identify a relevant problem to solve and confirm that the solution would be useful. This was followed by profiling our current approach (PLINK 1. 07) to understand what elements of the existing GWAS analysis process were rate limiting. Once the logistic regression was identified as the first element to tackle, we decomposed the problem by extracting the logistic regression code from PLINK to create a minimal code that served as the contest baseline.
A critical step at this point was definition of the test data to be used to score the contest. It is essential that test data accurately reflect the real-world data that the code will see in all relevant respects. This is important particularly because contestants will naturally optimize their submission using the specific test data that they are provided. This can often lead to lack of generalizability if test data sets do not capture the diversity of real-world datasets. Finally, we devised a scoring system that rewarded our most desired outcomes of speed and accuracy. The online platform we used offered different contest types. For the initial logistic regression contest we utilized a "Marathon Match. " Marathon matches with significant prizes can attract skilled participants, and the competitive orientation of these contests can deliver innovative and extreme value outcome solutions.
After the contest After the contest, the key steps we took included evaluation of the solutions in our HPC compute environment, review of the code, and merging, tracking, and supplementation of solutions over the course of multiple iterative contests. In general, these were standard scientific programming or software engineering activities that would occur in any software development project, but some elements were particularly salient in the crowdsourcing context. Within a single marathon match, contest codes were written and sometimes optimized for the contest hardware/OS/compiler environment. An ideal setup would have ensured the contest environment was identical to the intended platform for final use of the code, but in practice that was not always possible. For example, our HPC environment could not be provided to the contestants directly.
Hence, we found that the final speed and performance characteristics of codes in our environment were not always identical to the contest ranking. For example, some fast logistic regression codes used specific tricks or data structures that were either limiting or not performance enhancing in our environment. Hence, reviewing and benchmarking codes in our compute environment was essential. Across multiple, serial competitions, it was necessary to select best codes from an initial contest, possibly supplement them (e. g. , interfacing to PLINK format input data), and then supply the modified codes as input to a subsequent contest. In at least one case, we found a participant in one contest might reverse or remove code elements that were desirable for the overall project in order to maximize performance on their particular subproblem.
Conclusion Using iterative, competition-based OI, we have substantially accelerated logistic regression for GWAS analysis. The accelerated logistic regression code was donated and incorporated and is currently available in the PLINK2 open-source project  [26, 31]  to make it broadly available to the computational biology community, where it can enable the analysis of increasingly complex phenotype-genotype datasets. r Other Requirements: Codes have been tested on Red Hat En- terprise Linux 6 with gcc 4. 4. 7, 32 GB RAM. r License: GPLv2.
Acknowledgements We would like to thank the  Pfizer Business Technology High Performance Compute group  for their assistance with this project. We would like to thank  Sally John  and  Christoph Brockel  for providing guidance and thoughtful discussions. We also thank the members of the  Topcoder community  for participation in our contest and in particular the winning contestants, doudouille, allegro, zaq1xsw2tktk, syg96, venco, and  klo86min.
These images have width n = 1624 pixels and height m = 1232 pixels and 1 cm 2 is considered equal to 28,900 pixels. Phenospeed automatically computes the total projected rosette area (in cm 2 ). It cannot, however, computes the individual leaf areas necessary to exploit an organ-scale plant model. The dataset considered in this article consists of a series of T = 21 images for one plant of each of the 4 genotypes Burren (Bur), Columbia (Col), Shahdara (Sha), and Tsushima (Tsu). The plants were all grown in the same environmental conditions. The photoperiod was of 8 h, with a radiation of 350µmol m -2 s -1. The temperature was set to 21 • C during the day and 18 • C at night. The hygrometry was maintained constant at 65%RH. The series is composed of images taken on consecutive days from the 9th day after sowing (the day when the plants are installed on the robot) to the 29th day after sowing although, for the sake of clarity, the days of the image series will be identified from 1 to 21 in the following.
Data Analysis To better understand the dynamics of the whole plant, a series of measurements was performed on each image for the 4 genotypes available. Images are considered as elements of M m,n (R). Each point P of an image can be defined as an ordered pair P = (i, j) ∈ P, where P = [[1, m]] × [[1, n]], representing its row and column. On day t ∈ [[1, T]], an image has a set of n t visible leaves L t = {ℓ tu |u ∈ [[1, n t ]]}, where ℓ tu ⊂ P is referred to as a leaf segment (or segment for short) and is a connected subset of the image I t ∈ M m,n (R). Over the whole timeline [[1, T]], the plant has a set of N leaves L = {L v |v ∈ [[1, N]]} indexed by their order of appearance. A particular leaf of the plant is therefore identified by as many occurrences as images in the series, L v = {ℓ tv |t ∈ [[1, T]] , ℓ tv ∈ L t ∪ ∅}.
Automatic Leaf Segmentation The objective of this part is to search for all possible segments of leaves and their corresponding areas on each image of the series. The segmentation problem has been approached in various ways, with recent contributions using ellipsoid leaf-shape models  (Aksoy et al. , 2015) , Gaussian process shape models under a Bayesian approach  (Simek and Barnard, 2015)  or machinelearning  (Pape and Klukas, 2015). The approach used here was inspired from  Apelt et al. (2015). Some of the images provided by the Phenoscope software still contained objects from the background. Therefore, connected-segment labeling was first used to discard such objects not belonging to the plant, considered to be the main connected subset. the plant is then computed, as it constitutes, once artifacts have been removed, a very good approximation of the stem location from where the leaves grow.
Then a Canny edge detection filter  (Canny, 1986 ) is applied to help detect strongly overlapping leaves, before computing the Euclidean distance transform of the plant. The local maxima of this transform are searched, as they are the points the furthest from the background, and are therefore likely to correspond to mass center of leaves. They are hence used as markers for a watershed-based segmentation, which returns a set of connected segments susceptible to be leaves. The image processing operations were all performed using Python 3. 4. 3, and the library scikit-image 0. 12. 3. This first step of the segmentation returns a set of segments L(1) t = { l(1) tu |u ∈ 1, ñt }, which is different from the true set of segments found manually L t = {ℓ tu |u ∈ [[1, n t ]]}. The main difficulty of segmenting the leaves of a plant classically owes to the fact that some leaves might partially overlap some others, hence leading to segments of the resulting image to be (i) either only parts of a leaf (ii) or several distinct leaves merged.
The evaluated leaf areas will be used to estimate the parameters of the GreenLab functional-structural plant model presented in the next section, by model inversion. For a given leaf, only conserving segments in which there is a very high level of confidence might seem overly cautious. However, since the architecture is in itself representative of the whole plant functioning, there is a lot of redundant information contained in leaf area at close enough times. The daily image acquisition of the Phenoscope is thus in excess with respect to the time scale of A. thaliana source-sink dynamics. Overall, all the accepted points have a very low error compared to the true values and the amount of very precise data this method yields for a given plant (more than 60 values for each genotype) will prove sufficient for an accurate parameter estimation of the model.