dataesr/scientific-paragraphs-categorization huggingface dataset:
Introduction Preference data are usually collected when individuals are asked to rank a set of items according to a certain preference criterion. The booming of internet-related activities and applications in recent years has led to a rapid increase of the amount of data that have ranks as their natural scale, however often in the form of partial or indirect rankings (pairwise preferences, ratings, clicks). The amount of readily available software handling preference data has consequently increased consistently in the last decade or so, but not many packages are flexible enough to handle all types of data mentioned above. The typical tasks when analyzing preference data are rank aggregation, classification or clustering, and prediction, where the latter task refers to the estimation of the individual rankings of the assessors (when completely or partially missing).
These tasks can be addressed either via model-based inference or via heuristic machine learning algorithms, and (in the former case) with or without uncertainty quantification. However, not many methods allow combining various inferential tasks and individualized predictions. The R (R Core Team 2018a) package BayesMallows for analyzing ranking data, available at the Comprehensive R Archive Network (CRAN) at https://CRAN. R-project. org/, is one of the first attempts in this direction: BayesMallows implements full Bayesian inference for ranking data, and performs any of the tasks above in the framework of the Bayesian Mallows model (BMM)  (Mallows 1957; Vitelli, Sørensen, Crispino, Frigessi, and Arjas 2018). More specifically, BayesMallows allows for data in the forms of complete rankings, partial rankings, as well as possibly inconsistent pairwise comparisons.
Finally, BayesMallows implements the IPFP algorithm for computing the partition function for the Mallows model (MM)  (Mukherjee 2016 ) and the importance sampling algorithm described in  Vitelli et al. (2018). In addition to being used in the MM, these functions may be of interest in their own right. Comparing with other available inferential software, we notice that not many packages allow for such flexibility, very few in combination with full Bayesian inference, and none when using the MM. Often machine learning approaches focus on either rank aggregation (i. e. , consensus estimation), or individual rank prediction, while BayesMallows handles both. Since the BMM is fully Bayesian, all posterior quantities of interest are automatically available from BayesMallows for the first time for the MM.
In addition, the package also has tools for visualizing posterior distributions, and hence, posterior quantities as well as their associated uncertainty. Uncertainty quantification is often critical in real applications -for recommender systems, the model should not spam the users with very uncertain recommendations. When performing subtype identification for cancer patients, a very uncertain cluster assignment might have serious consequences for the clinical practice, for example in treatment choice. The package also works well with a fairly large number of items, thanks to computational approximations and efficient C++ programming. In conclusion, BayesMallows provides the first fully probabilistic inferential tool for the Mallows model with many different distances. It is flexible in the type of data it handles, and computationally efficient.
We therefore think that this software tool will gain popularity, and prove its usefulness in many practical situations, many of which we probably cannot foresee now. The paper is organized as follows. The BMM for ranking data is briefly reviewed in Section 2, as well as its model extensions to different data types and to mixtures. Section 3 includes details on the implementation of the inferential procedure. For a thorough discussion of both the model and its implementation we refer the interested reader to  Vitelli et al. (2018). An overview of existing R packages implementing the Mallows model (MM) is given in Section 4. The use of the BayesMallows package is presented, in the form of a quick-start guide, in Section 5. Section 6 concludes the paper, also discussing model extensions that will come with new releases of the package.
Background: The Bayesian Mallows Model for Rankings In this section we give the background for understanding the functions in the BayesMallows package. More details can be found in  Vitelli et al. (2018)  and  Liu, Crispino, Scheel, Vitelli, and Frigessi (2019). The section is organized as follows: we first clarify the notations that we will use throughout the paper (Section 2. 1). We then briefly describe the BMM for complete ranking data (Section 2. 2), also focusing on the relevance of the choice of distance (Section 2. 3). The last two sections focus on model extensions: partial and pairwise data (Section 2. 4), non-transitive pairwise comparisons (Section 2. 5), and mixtures (Section 2. 6).
Distance Measures and Partition Function The reason why the partition function Z n (α) reported in (1), (  2 ) and (4) does not depend on the latent consensus ranking ρ is that the chosen distance d(•, •) is right-invariant. A rightinvariant distance  (Diaconis 1988 ) is unaffected by a relabelling of the items. Right-invariant distances play an important role in the MM, and for this reason the BayesMallows package only handles right-invariant distances. The choice of distance affects the model fit to the data and the results of the analysis, and is crucial also because of its role in the partition function computation. Some right-invariant distances allow for analytical computation of the partition function, and for this reason they have become quite popular. In particular, the MM with Kendall (Meilǎ and Chen 2010; Lu and Boutilier 2014), Hamming  (Irurozki, Calvo, and Lozano 2014a)  and Cayley  (Irurozki, Calvo, and Lozano 2018)  distances have a closed form of Z n (α) due to  Fligner and Verducci (1986).
There are however important and natural right-invariant distances for which the computation of the partition function is NP-hard, in particular the footrule (l 1 ) and the Spearman (l 2 ) distances. For precise definitions of all distances involved in the MM we refer to  Marden (1995). The BayesMallows package handles the footrule, Spearman, Cayley, Hamming, Kendall, and Ulam distances. In the case of Cayley, Hamming, Kendall, and Ulam the exact partition function is implemented. In the case of footrule and Spearman there are three possibilities supported by the package: (i) for moderately large values of n, an exact implementation is available, and the maximum value of n for which this is available depends on the chosen distance; (ii) for larger values of n, an importance sampling approximation scheme is proposed, and the approximated Z n (α) has to be computed off-line over a grid of α values; (iii) for very large n, the asymptotic approximation of Mukherjee (  2016 ) is implemented, which also requires off-line computation.
Details on the MCMC Procedures In order to obtain samples from the posterior density of equation (  4 ), the BayesMallows package implements an MCMC scheme iterating between (i) updating ρ and (ii) updating α (Algorithm 1 of  Vitelli et al. 2018). The leap-and-shift proposal distribution, which is basically a random local perturbation of width L of a given ranking, is used for updating ρ in the (i) step. The L parameter of the leap-and-shift proposal controls how far the proposed ranking is from the current one, and it is therefore linked to the acceptance rate. The recommendation given in  Vitelli et al. (2018)  is to set it to L = n/5, which is also the default value in BayesMallows, but the user is allowed to choose a different value. For updating α in step (ii), a log-normal density is used as proposal, and its variance σ 2 α can be tuned to obtain a desired acceptance rate.
Partition Function As already mentioned in Section 2. 3, when the distance in the BMM is chosen to be the footrule or the Spearman, the partition function Z n (•) does not have a closed form. Hence, in these situations BayesMallows allows for three different choices, which the user may employ depending on the value of n: (a) exact calculation, (b) Importance Sampling (IS), and (c) the asymptotic approximation due to  Mukherjee (2016). The package contains integer sequences for the exact calculation of the partition function with footrule distance for up to n = 50 items, and with the Spearman distance for up to n = 14 items (see  Vitelli et al. 2018, Section 2. 1). These sequences are downloaded from the On-Line Encyclopedia of Integer Sequences  (Sloane 2017 ). The IS procedure can be used to compute an off-line approximation Ẑn (α) of Z n (α) for the specific value of n which is needed in the application at hand.
Packages Implementing the Mallows Model This section gives an overview of the existing R packages for fitting the MM. We refer to the supplementary online material for R scripts illustrating the use of these packages and tests supporting our conclusions. • PerMallows  (Irurozki, Calvo, and Lozano 2016)  is the package that comes closest in functionality to BayesMallows. It contains functions for learning and sampling from the frequentist versions of the MM and generalized Mallows model (GMM)  (Fligner and Verducci 1986). Compared to BayesMallows, it lacks support for footrule or Spearman distance, and it is not a Bayesian method, so it does not compute uncertainty ranges for the estimated parameters. In addition PerMallows handles only complete rankings, and does not provide functionality for computation of mixture models.
The function dbm returns the MLE of α together with its variance. The MLE of ρ, however, is not returned, but printed to the console, and no uncertainty estimates are given. Internally, dbm generates a matrix of sixe n!×n containing all possible permutations of the n items. As a result, it quickly runs into memory issues. In our tests, pmr was not able to handle a ranking dataset with n = 10 items. • rankdist (Qian 2018) implements distance-based probability models for ranking data as described in  Alvo and Yu (2014) , returning MLEs for α and ρ, but no uncertainty estimates. The package handles a large number of distances and supports mixture models, but in our experiments a warning was issued when using mixtures with all distances except Kendall. rankdist also implements the GMM  (Fligner and Verducci 1986).
R> install. packages("BayesMallows") The package is loaded into the current R session as follows: R> library("BayesMallows") An overview of the most used functions is provided in Table  1 , and Table  2  lists example datasets included in the package. The main function of BayesMallows is compute_mallows, which computes the posterior distribution of the BMM. Its arguments are listed in Table  3. The following sections demonstrate BayesMallows with example use cases.
Analysis of Complete Rankings We illustrate the case of complete rankings with the potato datasets described in  Liu et al. (2019, Section 4). In short, a bag of 20 potatoes was bought, and 12 assessors were asked to rank the potatoes by weight, first by visual inspection, and next by holding the potatoes in hand. These datasets are available in BayesMallows as matrices with names potato_weighing and potato_visual, respectively. The true ranking of the potatoes' weights is available in the vector potato_true_ranking. The rankings given to potatoes 1-10 by assessors 1-4 in the visual experiment is printed below. R> In general, compute_mallows expects ranking datasets to have one row for each assessor and one column for each item. Each row has to be a proper permutation, possibly with missing values.
We illustrate with the beach preference data containing stated pairwise preferences between random subsets of 15 images of beaches, by 60 assessors  (Vitelli et al. 2018, Section 6. 2 ). This dataset is provided in the dataframe beach_preferences, and we can show the first 10 rows with the command R> head(beach_preferences) Transitive Closure and Initial Ranking We start by generating the transitive closure implied by the pairwise preferences.
R> beach_init_rank <-generate_initial_ranking(beach_tc) If we had not generated the transitive closure and the initial ranking, compute_mallows would do this for us, but when calling compute_mallows repeatedly, it may save time to have these precomputed and saved for future re-use. In order to save time in the case of big datasets, the functions for generating transitive closures and initial rankings from transitive closures can all be run in parallel, as shown in the examples to the compute_mallows function. The key to the parallelization is that each assessor's preferences can be handled independently of the others, and this can speed up the process considerably with large dataset. As an example, let us look at all preferences stated by assessor 1 involving beach 2. We use filter from dplyr  (Wickham, François, Henry, and Müller 2018)   In addition to the statement that beach 15 is preferred to beach 2, all the other orderings stated by assessor 1 imply that this assessor prefers beach 6 to beach 2.
Convergence Diagnostics As with the potato data, we can do a test run to assess the convergence of the MCMC algorithm. However, this time we provide the initial rankings beach_init_rank to the rankings argument and the transitive closure beach_tc to the preferences argument of compute_mallows. We also set save_aug = TRUE to save the augmented rankings in each MCMC step, hence letting us assess the convergence of the augmented rankings. R> bmm_test <-compute_mallows(rankings = beach_init_rank, + preferences = beach_tc, save_aug = TRUE) Running assess_convergence for α and ρ shows good convergence after 1000 iterations (not shown). To check the convergence of the data augmentation scheme, we need to set parameter = "Rtilde", and also specify which items and assessors to plot. Let us start by considering items 2, 6, and 15 for assessor 1, which we studied above.
Clustering BayesMallows comes with a set of sushi preference data, in which 5000 assessors each have ranked a set of 10 types of sushi  (Kamishima 2003). Here are the first six rows of the dataset: It is interesting to see if we can find subsets of assessors with similar preferences. The sushi dataset was analyzed with the BMM by  Vitelli et al. (2018) , but the results in that paper differ somewhat from those obtained here, due to a bug in the function that was used to sample cluster probabilities from the Dirichlet distribution. R>.
Convergence Diagnostics BayesMallows has a convenient function compute_mallows_mixtures for computing multiple Mallows models with different numbers of mixture components. The function returns a list of models of class BayesMallowsMixtures, in which each list element contains a model with a given number of mixture components. Its arguments are n_clusters, which specifies the number of mixture components to compute, an optional parameter cl which can be set to the return value of the makeCluster function in the parallel package (R Core Team 2018a), and an ellipsis (. ) for passing on arguments to compute_mallows. Hypothesizing that we may not need more than 10 clusters to find a useful partitioning of the assessors, we start by doing test runs with 1, 4, 7, and 10 mixture components in order to assess convergence.
BayesMallows ignores label switching issues inside the MCMC, because it has been shown that this approach is better for ensuring full convergence of the chain  (Jasra, Holmes, and Stephens 2005; Celeux, Hurn, and Robert 2000). MCMC iterations can be reordered after convergence is achieved, for example by using the implementation of Stephens' algorithm  (Stephens 2000)  provided by the R package label. switching (Papastamoulis 2016). A full example of how to assess label switching after running compute_mallows is provided by running the following command: R> help("label_switching") For the sushi data analyzed in this section, no label switching is detected by Stephen's algorithm.
Discussion In this paper we discussed the methodological background and computational strategies for the BayesMallows package, implementing the inferential framework for the analysis of preference data based on the Bayesian Mallows model, as introduced in  Vitelli et al. (2018). The package aims at providing a general probabilistic tool, capable of performing various inferential tasks (estimation, classification, prediction) with a proper uncertainty quantification. Moreover, the package widens the applicability of the Mallows model, by providing reliable algorithms for approximating the associated partition function, which has been the bottleneck for a successful use of this general and flexible model so far. Finally, it handles a variety of preference data types (partial rankings, pairwise preferences), and it could possibly handle many others which can lie in the above mentioned categories (noisy continuous measurements, clicking data, ratings).
One of the most important features of the BayesMallows package is that, despite implementing a Bayesian model, and thus relying on MCMC algorithms, its clever implementation makes it actually possible to use it for large datasets. The package can easily handle up to hundreds of items, and thousands of assessors; an example is the Movielens data analyzed in Section 6. 4 of  (Vitelli et al. 2018). By using the log-sum-exp trick, the implementation of the importance sampler is able to handle at least ten thousand items without numerical overflow. We believe that all these features make the package a unique resource for fitting the Mallows model to large data, with the benefits of a fully probabilistic interpretation. Nonetheless, we also recognize that the BayesMallows package can open the way for further generalizations.
The Bayesian Mallows model for time-varying rankings that has been introduced in  Asfaw, Vitelli, Sørensen, Arjas, and Frigessi (2017)  will be considered for a future release. Some further extensions which we might consider to implement in the BayesMallows in the future include: fitting an infinite mixture of Mallows models for automatically performing model selection; allowing for a non-uniform prior for ρ; performing automatic item selection; estimating the assessors' quality as rankers; and finally including covariates, both on the assessors and on the items. In addition, since the data augmentation steps in the MCMC algorithm are independent across assessors, potential speedup in the case of missing data or pairwise preferences can be obtained by updating the augmented data in parallel, and this is likely to be part of a future package update.
Computational Details The M-H algorithm for computing the BMM, as well as the importance sampling algorithm for estimating the partition function, are implemented in C++, heavily utilizing the Armadillo library for linear algebra and scientific computing  (Sanderson and Curtin 2016, 2018)  The devtools package provided very useful tools during package development, and the function check_with_sanitizers function in the rhub package (CsÃąrdi 2017) was useful for detecting memory issues in the C++ code using address sanitizers and leak sanitizers as described in Sections 4. 3. 3 and 4. 3. 4 of Writing R Extensions R Core Team (2018b).
By exchanging Bait and Prey for a given couple it is possible to probe the corresponding PPI in two different orientations. It is known that the two orientations may lead to different read-outs  (13). As a second filter (after the auto-activation filter), we measured therefore for each couple the reporter level for both orientations (see Suppl. Fig. S2  & Suppl. Data 1). Table  1  lists the couples in the orientation with the stronger reporter level (when the cellular contents of BD-Bait and AD-Prey are standardized, see below). This orientation is considered as the molecular configuration with the higher accessibility of the PPI binding interface  (13). In other words, this orientation may feature the smaller steric hindrance due to the fused fluorescent BD and AD and probably resembles more closely the situation of free (not fused) proteins.
Data analyses The flow-cytometry files were analyzed using the FlowCytometryTools package for python (http://eyurtsev. github. io/FlowCytometryTools). For visualization purposes, the hlogtransformation (47) was applied to the signals of the channels TagRFP-H, EGFP-H, TagBFP-H with the following settings: b = 1000, r = 10000, and d = 5. 4. The transformed values were then visualized as 2-dimensional scatter plots using the built-in function of FlowCytometryTools (Fig. 2 ) or as 1-dimensional probability distribution functions (PDFs) using the package matplotlib (Fig. 4 ). The PDFs correspond to histograms generated with the package NumPy (using 50 bins), normalized so the integral over the range -1000 to 10000 is 1. To quantify the reporter level, we calculated the mean value of the Tag-BFP-H channel, <Tag-BFP-H>, either for the entire population of cells or for double-gated subpopulations.
1 where Ntot is the total number of events within the data range 0-50000 for the given sample. For li→infinity, the cumulative mean approaches the population mean but it remains slightly smaller due to binning issues. We provide a Python based program with a graphical user interface that automates the calculation of the cumulative mean for large datasets (see section data availability). With this program, the analysis can also be performed with double-gated subpopulations.
The magnitude of the reporter level varied between different repetitions of the experiment. This is probably due to differences in the transformation efficiency between different batches of competent yeast cells (see Suppl. Fig. S6A ). Differences in transformation efficiency may lead to different copy number of the reporter plasmid. By applying the following two-point normalization, these variations could be attenuated (Suppl. Fig. S6B ): Eq. 5 By definition, the normalized mean of the CTRL sample (=background of the system) is set to 0% when the normalized mean of the couple with the highest affinity, i. e. , BD-Barstar Y29F / AD-Barnase H102A, is set to 100%. This normalization procedure can be performed with our Pythonbased program (see Data availability section). To test the null hypothesis that the normalized mean of a given couple is larger than the normalized mean of another couple (with lower affinity, Fig.
5 man-month. This throughput is based on our experience during the setup phase of the qY2H approach when we tested different experimental conditions. Thus, about 40 couples can in principle be characterized (including the auto-activation and orientation filters and five repetitions of the experiments) within one month by a single manipulator. The throughput of all steps of the protocol (except data acquisition by flow cytometry) can be scaled linearly with increasing manpower. As these steps have been optimized for liquid phase, a scale-up with robots could even be envisaged  (61). The acquisition by flow cytometry might be also a limiting step for the scalability. With the MACSQuant VYB flow cytometer used in this work, a maximum of 100 samples can be measured per day (with a flow speed of 16000 events/s).
The topology of force-directed networks may help identifying key pathways within the network, and how these paths change as a function of environmental conditions (stress, metabolism, etc). Thus, we anticipate that qY2H data would boost the modelling of interactomes and thereby advance significantly systems biology. Tables  Table 1:  List of BD-Bait and AD-Prey couples tested with the qY2H approach. A set of known interactions was generated with proteins pairs with a reported affinity. The dissociation constants (Kd) were taken from the referenced literature and are given in pM; the molecular weights (MW) are in kDa. An additional Specificity Test Set was composed by cross-testing a sub-selection of BD-Bait and AD-Preys fusions of the Affinity Test Set, complemented with the fusions BD-ARMC1, BD-GMPPA, AD-Emerin and AD-MNAT1.