dataesr/scientific-paragraphs-categorization huggingface dataset:
Data Model Workers: We have a set U = {u1, u2,. , un} of n available workers. For the NBA application, workers are players, whereas in the co-authorship application, workers are the authors. Domains & Skills: We are given a set of skill domains D = {d1, d2,. , dm}. Skill domains are associated with both tasks and workers. We assume that each domain is independent and focus on estimating workers' skill per domain. For the NBA application, domains could be defense or attack and each player has a value for each domain (0 denotes no skill in a domain). Our problem now simplifies to estimating a single skill per worker and invoking the estimation for each domain independently. We represent a worker skill as s u. The skills of all n workers are presented by a vector -→ S. s u is either a deterministic value, or a probability distribution.
This process continues until no change can be found to improve the error. The solution is then said to be "locally optimal". With random restart, the algorithm performs hill-climbing iteratively, each time with a random initial condition and the best solution is kept at the end as the final solution. The various restarts increase the likelihood of finding "global optima". Our algorithm discovers the global optima on Example 1 and produces the following 3 distributions, s u 1 = [0, 0, 1], s u 2 = [1, 0, 0], s u 3 = [0, 1, 0]. Complexity: The exact asymptotic form is hard to derive, as that depends on how fast it reaches the local optima in a given iteration. Our experimental results indicate that the solution converges within a few minutes most of the time even for a large scale dataset.
Dataset Descriptions 1) NBA: We collected 317, 371 tuples of NBA scores from 1991-2004 regular season. We pre-process this dataset and generate the worker to task assignment matrix A by matching players with games. We consider two independent skill dimensions, i) Number of points, ii) Number of Assists where the team skill is the computed by the additive skill model. Our final dataset contains 21000 matches and 1200 players. Ground truth in NBA dataset: The ground truth consists of the number of points and the number of assists of a player played in a particular game. If a player has played several games (which is really the case always), this gives rise to a distribution, as opposed to a single skill value per worker. 2) DBLP: We use a subset of DBLP  4  considering the papers that are published from year 2000.
We primarily consider authors who publish in database conferences (SIG-MOD, VLDB, CIKM, ICDE, EDBT), in the area of query processing. Each publication is a completed task that is undertaken by a set of authors; we consider the number of citations as its quality. This dataset consists of 20123 publications and 22700 unique authors. Ground truth in DBLP dataset: Unlike the NBA dataset, there is no ground truth available per worker (i. e. , no truth is known about an author's expertise). Therefore, we neither have a skill pdf nor a single skill value per worker. As we describe in Section 7. 3, we take up a cross-validation type of approach for evaluation here. 3) Synthetic Dataset: We generate a synthetic dataset for evaluating the scalability of our proposed solutions for the deterministic and probabilistic algorithms.
(2) Max-Skill-P : Our proposed solution in Section 5. 2 is compared with the baseline solution, whenever appropriate. [12]  to evaluate our deterministic algorithms. We divide the dataset in train and test and perform 3-fold cross validation. In particular, each record in a test set is a task that is undertaken by a set of workers. For each such worker, we estimate their respective skill considering only the train dataset (those who do not appear in train get a skill of 0). For each task in test set, the ground truth (i. e. , the true quality) is the associated quality value. We compare the estimated quality of a task with that of the ground truth for that task.
Relative error is computed as √ ∀t e t ×e t √ ∀t q t ×q t. We present the percentage of tasks in our test set which overestimates the task quality (compared to the ground truth), thus violating the one-sided error constraints (Section 3. 2. 1). Probabilistic Scenario: For the probabilistic variant of the NBA dataset, we compute the 2 error between the estimated skill pdfs and that of the ground truth distributions. We measure relative error over the test dataset (assuming the evaluated quality of each task in the test set as ground truth). We need to transform the estimated pdf of each worker to a single value by computing expected skill, i. e. , ExpectedSkill(u) = w (P r(s u = w) × w) and measuring the 2 error between the actual quality and the expected quality of the tasks. We discretize the pdfs using w equi-width histograms (buckets), where each bucket is a skill range.
Summary of Results Quality Experiments: Our first set of results (considering relative error) strongly corroborate our hypothesis on the underlying skill aggregation model for a given application -i. e. , NBA dataset follows Sum-Skill, whereas, DBLP dataset follows Max-Skill. After this result, we present the rest of the experiments by considering the most appropriate dataset for it (i. e. ,NBA for Sum, DBLP for Max). Our deterministic algorithms demonstrate that we consistently outperform all the baseline algorithms (including the regression based one) in minimizing the error value, as well as consistently obeying the one sided error constraints. Same observation holds for the probabilistic algorithms, where Sum-Skill-P and Max-Skill-P significantly outperform their respective baseline counterparts.
Scalability Experiments: For the deterministic scenario, our results indicate that our proposed solutions are scalable. Even when the task assignment matrix is very large, they only take a few minutes to complete. Max-Skill-D is more scalable than Sum-Skill-D. This is consistent with our theoretical analyses in Section 5. 1. For the probabilistic scenario, the heuristic algorithms are more efficient than the optimal variants and often converge within few minutes and scale well. For lack of space, we only present a subset of results. The presented results are representative. Deterministic/Probabilistic Skill Models: To highlight the tradeoffs between deterministic and probabilistic models, we conduct three experiments. We generate a synthetic dataset where each worker is associated with a skill pdf where we vary the variance for a fixed task assignment matrix.
Max-Skill We use the DLBP dataset to perform further experiments on Max-Skill. We vary the number of tasks and measure the average absolute 2 error. Figure  5 (a) shows that Max-Skill-D outperforms BL-Max-D algorithm. With increasing training set, the algorithm learns better, hence the error of Max-Skill-D decreases. Figure  5(b)  shows that Max-Skill-P outperforms the probabilistic baseline BL-Max-P consistently. However, error varies with the task size using Max-Skill-P , because the algorithm cannot always find the global optima. Figure  5 (c) shows the comparison between Max-Skill-D and Max-Skill-P. We observe that Max-Skill-D performs better for larger training data. However, Max-Skill-P provides the distribution of worker skill which can be of importance for a particular application.
Probabilistic Estimation: Sum and Max Skill For the probabilistic skill, for brevity, we only present a subset of results. As stated before, we vary the parameters that influence the efficiency of the hill climbing algorithms. Varying w -Max: As we increase the number of skill buckets, the number of unknown to solve increases, hence it takes longer to converge. Figure  7 (a) demonstrates that Max-Skill-P scales well with varying w for five different size datasets. Sum-Skill-P takes about 60% more time than Max-Skill-P as the joint pdf has larger ranges (e. g. , two pdfs in the range of [0 -15] gives a joint pdf between [0 -30] for sum, whereas, it is still [0-15] for max) for sum, thereby requiring more computations. Although efficiency decreases with increasing w, but we get the distribution of worker skill with more granularity.
CONCLUSION We initiate the study of estimating skill of the individual workers in team based tasks for various applications. We formalize it as an optimization problem considering multiple skill aggregation functions, where skill of a worker is either deterministic or a probability distribution. We propose principled solutions to all the studied variants of the problem and provide in-depth analyses. We run a comprehensive set of experiments considering two real world datasets that demonstrate the effectiveness of our proposed skill estimation algo-   rithms. We also conduct large scale synthetic experiments to validate the scalability of our proposed solutions.
The biotinylated cRNA was used for hybridization. The Affymetrix HWS Kit for GeneTitan (part nr. 901622) was used for hybridization, washing and staining of the array plates. The array plates were scanned using the Affymetrix Command Console v3. 2 software. All data are MIAME compliant as detailed on the MGED Society website http://www. mged. org/Work groups/MIAME/miame. html. The microarray data have been deposited on the NCBI's Gene Expression Omnibus  (Edgar et al. 2002)  and is accessible through the GEO Series accession number GSE62876.
Network construction and analysis Pearson correlation coefficients between all pairs of DEGs were calculated. A table with correlation coefficient values was exported to Cytoscape v. 2. 8. 2  (Smoot et al. 2011 ) and correlation coefficients above a threshold of 0. 97 (determined according to  Freeman et al. 2007)  were used to filter the connections that were used to determine the edges between nodes in the network. The resulting network was displayed with a yGraph Organic layout. The Cytoscape plug-in NetworkAnalyzer  (Assenov et al. 2008 ) was used to compute node degree. The 100 nodes with the highest degrees were considered ''hubs''. Self-organizing maps (SOMs) were calculated using GeneMaths software (version 2. 1, Applied Maths BVBA, Sint-Martens-Latem, Belgium) by importing gene expression data and mapping them into six groups (2 9 3 node format) that provided optimal representation of gene expression patterns in a reasonably small number of independent bins.
3 ). In total, 1177 genes were considered differentially expressed in response to the ABA treatment for at least one time-point as compared to the control. The number of DEGs was the lowest after 2 h of incubation and increased until 24 h, after which no substantial changes occurred until 72 h of incubation (Table  1 ). To verify the accuracy of the microarray data, the expression of 20 genes with different expression patterns were also analyzed by qPCR. Both microarray data and qPCR analysis showed comparable trends (Suppl. Fig. S2 ).
Incubation in ABA reduces abundance of transcripts related to expansins (ATEXPA1, ATEXPA8 and ATEXPA 10), extensins (AT2G43150, AT3G28550 and AT3G54580) and certain photosynthesis-related genes (APT3, APT5, GUN4, GUN5, HEMA1 and PORB) that are not significantly affected by PEG treatment, while PEG reduced the abundance of gene transcripts related to fatty-acid metabolism (AT5G10160, ACP4, HCD1, OPR1 and OPR2) and related to the synthesis of compounds that contain sulfur (ATGSTU27, CYP79B2, CYP83B1, MAT3 and MTO3), such as the amino acid cysteine. Considering that in plant genomes many TFs are master regulators of signaling and regulatory pathways of stress adaptation and act in an ABA-dependent manner (Lindemose et al. 2013), we searched for the presence of known and predicted TFs as listed in the database of Arabidopsis transcription factors (DATF, http://www.
Oleosins are typical for the later stages of seed development and, in desiccation tolerant seeds, they prevent lipid bodies from coalescing on dehydration and prevent the disruption of cellular structures during rehydration  (Leprince et al. 1998; Pammenter and Berjak 1999). Despite the fact that all the germinated seeds incubated in ABA had re-established DT after 24 h, the incubation was maintained for a total of 72 h, which improves longevity compared with incubation for 24 h (Fig. 2 ). Also during development, acquisition of DT precedes acquisition of longevity, likely to ensure timely response of stressrelated genes (such as LEAs, HSPs and oxidative stressrelated genes) that also improve seed storability  (Verdier et al. 2013). Transcriptomic studies generate large datasets. Visualization and analysis of this kind of data as networks is an important approach to explore a wide variety of biological relations  (Freeman et al.
Transcription Activator-Like (TAL) effectors from Xanthomonas plant pathogenic bacteria can bind to the promoter region of plant genes and induce their expression. DNA-binding specificity is governed by a central domain made of nearly identical repeats, each determining the recognition of one base pair via two amino acid residues (a. k. a. Repeat Variable Di-residue, or RVD). Knowing how TAL effectors differ from each other within and between strains would be useful to infer functional and evolutionary relationships, but their repetitive nature precludes reliable use of traditional alignment methods. The suite QueTAL was therefore developed to offer tailored tools for comparison of TAL effector genes. The program DisTAL considers each repeat as a unit, transforms a TAL effector sequence into a sequence of coded repeats and makes pair-wise alignments between these coded sequences to construct trees.
The program FuncTAL is aimed at finding TAL effectors with similar DNA-binding capabilities. It calculates correlations between position weight matrices of potential target DNA sequence predicted from the RVD sequence, and builds trees based on these correlations. The programs accurately represented phylogenetic and functional relationships between TAL effectors using either simulated or literature-curated data. When using the programs on a large set of TAL effector sequences, the DisTAL tree largely reflected the expected species phylogeny. In contrast, FuncTAL showed that TAL effectors with similar binding capabilities can be found between phylogenetically distant taxa. This suite will help users to rapidly analyse any TAL effector genes of interest and compare them to other available TAL genes and should improve our understanding of TAL effectors evolution.
Datasets The sequences of 229 TAL effectors were obtained from the NCBI protein and nucleotide databases (http://www. ncbi. nlm. nih. gov/; accession numbers in Supplementary Table  1 ). This set was used for all analyses unless indicated and is referred to as the public dataset. 496 additional sequences (awaiting publication) were provided by collaborating laboratories, including those reported in  Wilkins et al. (2015). These, together with public TAL effectors sequences are referred to as the full dataset. The species composition of the full dataset is found in Supplementary Table  2.
Program Specifications FuncTAL and DisTAL are implemented in the Perl and R programming languages, they use the Perl modules Statistics::R, Bio::Perl  (Stajich et al. , 2002) , and the R library APE  (Paradis et al. , 2004). DisTAL additionally uses the module Algorithm::NeedlemanWunsch (http://search. cpan. org/∼vbar/ Algorithm-NeedlemanWunsch-0. 03/lib/Algorithm/Needleman Wunsch. pm) to align repeats and the C++ program ARLEM version 1. 0  (Abouelhoda et al. , 2010)  to align sequences of coded repeats. Penalty parameters values for NeedlemanWunsch alignments are gap: 0, mismatch: -1, match: +1, alignment scores are normalized by dividing the score by the maximum length among the analyzed sequences and multiplying by 100 so they can be used by ARLEM. Parameters values for ARLEM are: align = TRUE, -insert = TRUE, ARLEM alignment scores are divided by 100 (so they can be used to build trees).
Neighborjoining trees are generated using the nj function of the package APE with default parameters  (Paradis et al. , 2004). The input file for DisTAL is a FASTA file containing amino acid sequences of TAL effectors. An additional file containing information on the TAL effectors can be used to color code the trees generated by the program. The following parameters can be modified: layout of the output tree (default = unrooted), include and compare input to TAL effectors from the public dataset (default = false), number of similar TAL effectors to output if the public database option is active (default = 5), exclude RVDs from analysis (default = false). Additional parameters can be modified in the standalone version: ARLEM indel penalization (default = 10), ARLEM duplication penalization (default = 10), and Create repeat distance matrix de novo (default = false).
The outputs generated by DisTAL are: • a pseudo-FASTA file (Outputname_CodedRepeats. fa) with the TAL effectors coded as a string of numbers, • the set of unique repeats in the input file and their number codes (Outputname_RepeatsCode. txt) • a matrix (Outputname. mat) showing the distances between the TAL effectors, • a tree file (Outputname. tre) in Newick format to be used in any tree visualization program, • a hits file (Outputname. hits) if the option to compare to a database was used; this file shows, for each TAL effector, the closest matches in our database. DisTAL took an average of 0 m 22. 3 s to process 200 TAL effector sequences in a computer with a Linux operating system with 15. 6 Gb of RAM and an Intel R Core ™ i7-4600U CPU @ 2. 10 GHz processor. When the option to compare against public TAL effectors is activated, time goes up to 0 m 35.
5 s. FuncTAL uses modified subroutines (readMotifFile, compMotifs, scoreComparison, correlation) from the program compareMotifs. pl from the Homer (Hypergeometric Optimization of Motif EnRichment) suite  (Heinz et al. , 2010). The program can take as an input a text file with RVD sequences in the format ">RVD_id<tab>HD-NN-HD. " or a FASTA file containing nucleotide or amino acid sequences of TAL effectors. If a FASTA file is entered, the program will first recognize repeats in the TAL effector sequence as described for DisTAL. For each repeat the program next extracts the RVDs, i. e. , the 12th and 13th amino acid (e. g. , NN-HD). If the 13th amino acid is missing, as is the case for some repeats, the program inserts an asterisk " *. " Neighbor-joining trees are generated using the nj function of the package APE with default parameters  (Paradis et al.
, 2004). The following parameters can be modified: layout of the output tree (default = unrooted), include and compare input to TAL effectors from the public dataset (default = false), and number of similar TAL effectors to output if the public database option is active (default = false). The outputs generated by FuncTAL are: • a text file (Outputname. cons) showing the theoretical most likely binding site for each TAL effector, • a matrix (Outputname. mat) showing the distances between the TAL effectors, • a tree file (Outputname. tre) in Newick format to be used in any tree visualization program, • a hits file (Outputname. hits) if the option to compare to a database was used, this file shows, for each TAL effector, the closest matches in our database according to their binding sites. FuncTAL took an average of 1 m 22.
3 s to process 200 TAL sequences in a computer with a Linux operating system with 15. 6 Gb of RAM and an Intel R Core ™ i7-4600U CPU @ 2. 10 GHz processor. When the option to compare against public TAL effectors is activated, time goes up to 7 m 48. 5 s. The script used for simulated evolution of TAL effectors (Evolve. pl) is also made available at http://sourceforge. net/ projects/quetaleffectors. This program uses the dist. topo function of the program APE  (Paradis et al. , 2004)  to calculate topological distances between trees using the Penny and Hendy method  (Penny and Hendy, 1985). The topological distance is defined as twice the number of internal branches defining different bipartitions of the tips  (Penny and Hendy, 1985). The distances were normalized by the number of nodes in a tree.
8. 31 with default parameters. Alignments for t-coffee  (Notredame et al. , 2000)  were made using version 10. 00. r1613 (-gapopen = -50, -gapext = 0). And MAFFT  (Katoh et al. , 2005)  alignments used the version 7. 123b (E-INS-i -ep 0 -genafpairmaxiterate 1000). Parameters were chosen to allow long gaps in alignments. Trees were generated from these alignments using the dist-align function from the "seqinr" R package (http:// seqinr. r-forge. r-project. org/) and the nj function from the APE package  (Heinz et al. , 2010).
Programs Availability Packages containing the scripts of the FuncTAL and DisTAL programs, as well as additional scripts used in this work are available for download from Sourceforge at http://sourceforge. net/projects/quetaleffectors. A web interface and the source code for the suite are also available at http://bioinfo-web. mpl. ird. fr/ cgi-bin2/quetal/quetal. cgi. The web version was created in Perl cgi-bin with w3c recommendations for CSS level 3 and Html 5. 0 http://www. w3. org/standards/webdesign/htmlcss.
, 1992; Boch et al. , 2009; Streubel et al. , 2012) , and that deletions or duplications occur in nature or and may be responsible for change in virulence and aggressiveness  (Vera Cruz et al. , 2000). To phylogenetically classify TAL effectors, we developed the program DisTAL, which classifies the input TAL effectors as a string of coded repeats and then uses the program ARLEM to calculate distances between these strings. The workflow for this program is depicted in Figure  1 , and described next in detail.
Identification and Coding of Repeats The program takes as input a set of TAL effectors to be analyzed, and if desired, the input TAL effectors can be compared to a dedicated database of 229 TAL effectors available in public DNA sequence databases (Supplementary Table  1 ). The input file should be a FASTA file containing either nucleotide or amino acid sequences of TAL effectors. If the input is nucleotide sequences these are translated to amino acids (in reading frame +1). It identifies and separates repeats in the input sequences by finding matches to motifs of 7 amino acids found at the start of repeats of known TAL effectors as traditionally defined  (Boch and Bonas, 2010) (i. e. , LTPDQVV). The program can also identify aberrant repeats (longer or shorter than average) and keep them for analyses.
Calculating Distances between Unique Repeats Next, a distance matrix is generated by calculating distances between every pair of unique repeats. For this, a global alignment with sliding ends (no gap penalty) is made for each pair of unique repeats using the Needleman-Wunsch algorithm  (Needleman and Wunsch, 1970)  as implemented in the Perl package Algorithm::NeedlemanWunsch (http://search. cpan. org/~vbar/Algorithm-NeedlemanWunsch-0. 03/lib/Algorithm/ NeedlemanWunsch. pm). The distances are normalized so that the repeat matrix information can be interpreted as the percentage of amino acids that change between repeats (based on the longest repeat among the two aligned). A distance matrix was generated for a set of 1110 unique TAL effector repeats found in our full dataset. It is included in the web version and the standalone version of the program to save computational time.
If new repeats are found in the input file these are compared to the existing matrix and added to it. Alternatively, the user can choose to generate this matrix using the Smith-Waterman  (Smith and Waterman, 1981)  algorithm for pairwise alignments as implemented in the Perl package Bio::Tools::pSW (http://search. cpan. org/dist/BioPerl/Bio/Tools/ pSW. pm) using different amino acid substitution matrices (PAM30, PAM50, and Blosum62). This strategy is so far only available in the standalone version and it has not been extensively tested, however the results obtained with either matrix are often similar; the average topological distances for 50 trees obtained from 10 randomly selected TAL effector sequences when comparing the trees obtained with the Needleman-Wunsch algorithm to those obtained using Smith-Waterman + PAM30, PAM50, and Blosum62 were 0.