Package: drrank
DRrankDRrank is a Python library to implement the Empirical Bayes ranking scheme developed inKline, Rose, and Walters (2023). This code was originally developed byHadar Avivi.Installation:The package uses the Gurobi optimizer. To useDRrankyou must first install Gurobi and acquire a license. More guidance is available from Gurobihere. Gurobi offers a variety of free licenses for academic use. For more information, see the followingpage.After having successfully set up Gurobipy, installDRrankvia pip:pipinstalldrrankUsage1. Load sample dataDRrankgrades units based on noisy estimates of a latent attribute. You can construct these estimates however you'd like---allDRrankrequires is a vector of estimates, $\hat{\theta}_i$, and their associated standard errors, $s_i$.To illustrate the package's features, this readme uses the data inexample/name_example.csv, which contains estimates of name-specific contact rates from the experiment studied in Kline, Rose, and Walters (2023). These contact rates have been adjusted to stabilize their variances using the Bartlett (1936) transformation. Variance-stabilization is useful because the deconvolution procedure used in Step 2 below requires that $s_i$ be independent of $\theta_i$. In cases where variance stabilization is not possible, independence can sometimes be restored by residualizing $\hat{\theta_i}$ against $s_i$; see Section 5 ofKline, Rose, and Walters (2023)for a detailed example. The transformation used in our names example computes estimates as $\hat{\theta}_i = sin^{-1} \sqrt{\hat{p}_i}$, where $\hat{p}_i$ is share of applications with name $i$ that received a callback. As discussed in the paper, $\hat{\theta}_i$ has asymptotic variance of $(4N_i)^{-1}$, where $N_i$ is the number of applications sent with name $i$.importpandasaspdimportos# Read in the datadata=pd.read_csv(os.getcwd()+'/example/name_example.csv')data.head()name_idthetassfirstname10.537880.0137214Adam20.5183940.0173332Aisha30.537670.0174933Allison40.5321290.0138126Amanda50.5349980.0136503AmyWhileDRrankprovides the functionality to account for any variance-stabilizing transformation, using one is not strictly necessary.DRrankcan also accomodate cases where the $\hat{\theta}_i$ directly capture untransformed estimates of the relevant latent attribute.2. Estimating the priorDRrankprovides the functionality to estimate a prior distribution using a variation on Efron (2016)'slog-spline deconvolutionapproach, which uses an exponential family mixing distribution with density parameterized by a flexible B-th order natural  cubic spline.To estimate the prior, generate an instance of theprior_estimateclass with each unit's estimated latent attribute, $\hat{\theta}_i$, and its associated standard errors. You also have the option of supplying an inverse transform in case the $\hat{\theta}_i$ have been transformed to stabilize variances. The appropriate inverse transform for the ranking name-specific contact rates in Kline, Rose, and Walters (2023), for example, is $f(x) = sin(x)^2$. The inverse transform function should be vectorized.importnumpyasnpfromdrrank_distributionimportprior_estimatethetas=data.thetas.values# set of estimatess=data.s.values# setstandard errors# Initialize the estimatorG=prior_estimate(thetas,s,transform=lambdax:np.power(np.sin(x),2))# Estimate the prior distributionG.estimate_prior(supp_points=5000,spline_order=5,seed=123)Use thesupp_pointsoption (default=5000) to pick the number of points of support over which to evaluate the prior density. The minimum and maximum of the support are the minimum and maximum ofthetas.Use thespline_orderoption (default=5) to adjust the degrees of freedom of the spline that parameterizes the mixing distribution.The estimated prior distribution will be saved as a dictionary. You can access it with the following code:# Dictionary with the prior distributionG.prior_g# Keys:# mean_theta: mean of the priorG.prior_g['mean_theta']# sd_theta: standard deviation of the priorG.prior_g['sd_theta']# g_theta: estimated prior densityG.prior_g['g_theta']You can then graph the results by calling the following function:# Plot the estimated prior distributionG.plot_estimates(save_path="example/prior_distribution.jpg",binwidth=0.0030,line_kws={'alpha':0.6},fill=True,alpha=0.3)Within the function you can specify the following arguments:g_theta: provide your own prior distribution G.Noneimplies the function will utilize the estimated G from theestimate_prior()method (default =None).show_plot: whether to show the plot or not (default =True).save_path: path to where the plot will be saved.Noneimplies the graph will not be saved (default =None).You can also furtherly change the visualization of the histogram, which inherits all the arguments fromseaborn.histplot3. Estimation of posterior features and $P$ matrixOnce the prior distribution has been estimated, you can estimate posterior means and credible intervals, as well as the matrix of pairwise posterior ordering probabilities $P$.# Compute the posterior featuresG.compute_posteriors(alpha=.05,g_theta=None)G.pmean# posterior meansG.pmean_trans# inverse transformed posterior meansG.lci# lower limit of 1-alpha credible intervalG.uci# upper limit of 1-alpha credible intervalG.lci_trans# lower limit of inverse transformed 1-alpha credible intervalG.uci_trans# upper limit of inverse transformed 1-alpha credible intervalG.posterior_df.head()# Dataframe of posterior featurespmeanpmean_translciucilci_transuci_trans00.5225240.2490880.507710.5321690.2363680.25745810.5150320.2426780.4931660.5296460.2241240.25525520.520510.2473610.4974250.5315420.2276860.2569130.5212650.2480050.5009280.5314460.230630.25682640.5219930.2486310.504560.5318150.2336970.257149Then compute the pairwise ordering probabilities $P$ using:# Compute the pairwise ordering probabilitiespis=G.compute_pis(g_theta=None,ncores=-1,power=0)In both functions, it is possible to provide your own prior distribution G by feeding an array as theg_thetaargument. This density must take support on the values determined byG.supp_theta.compute_pisalso provides the option to compute the elements of $P$, $\pi_{ij}$, as the posterior expectation of $max(\theta_i - \theta_j,0)^{power}$, providing an extension to weighted ranking exercises. The default, $power=0$, will produce $\pi_{ij}$ that are posterior ordering probabilities discussed in the next section and implies that ranking mistakes a considered equally costly regardless of the cardinal difference between $\theta_i$ and $\theta_j$.4. Computing gradesTo compute rankings, use thefitfunction with a matrix $P$. In the unweighted case, $P$ reflects the posterior probabilities that observation i's latent measure $\theta_i$ exceeds unit j's. That is, each element of this matrix takes the form:$\pi_{ij} = Pr(\theta_i > \theta_j | Y_i = y_i, Y_j = y_j)$DRrankexpects these probabilities to satisfy $\pi_{ij} = 1-\pi_{ji}$ and will report a warning if that does not appear to be the case.In the weighted case, $P$ captures expected differences between i and j's value of $\theta$, as discussed above.There are two ways to useDRrank.First, one can supply a parameter $\lambda \in [0,1]$, which corresponds to the user's value of correctly ranking pairs of units relative to the costs of misclassifying them. $\lambda=1$ implies correct and incorrect rankings are valued equally, while $\lambda=0$ implies correct rankings are not valued at all. In the unweighted case, it is optimal to assign unit $i$ a higher grade than unit $j$ when $\pi_{ij} > 1/(1+\lambda)$, which implies $\lambda$ also determines the minimum level of posterior certainty required to rank units pairwise.fromdrrankimportfit# Fit the report card functionresults=fit(pis,lamb=0.25,DR=None)The results object contains the row index of $P$, the assigned grades, and the Condorcet rank (i.e., grade under $\lambda=1$).obs_idxgrades_lamb0.25condorcet_rank11321323112419515We also provide functionality to compute results for a list of values of $\lambda$ in parallel:importnumpyasnpfromdrrankimportfit_multiple# Try different values of Lambdaresults_l=fit_multiple(pis,np.arange(0,0.9,0.01))Second, one can askDRrankto compute grades that maximize Kendall (1938)'s $\tau$, a measure of the rank correlation between units' latent rankings and assigned grades, subject to a constraint on the expected share of pairwise units incorrectly classified, which we refer to as the Discordance Rate (DR).# Fit the report card functionresults_dr=fit(pis,lamb=None,DR=0.05)Finally, we provide functionality to plot grades along with posterior means and credible intervals:fromdrrankimportfig_ranks# Merge the results with the identity of our observationsresults['firstname']=data.firstnamefig_ranks(ranking=results,posterior_features=G.posterior_df,ylabels='firstname',ylabel_fontsize=8,save_path='example/name_ranking.jpg')Within the function you can specify the following arguments:results: ranking results fromdrrank.fitposterior_features: posterior features computed throughG.compute_posteriors()ylabels: optional, specify the column inresultswhere we have stored the labels of each observation (default =None)ylabel_fontsize: optional, specify font size for labels (default = 8)show_plot: whether to show the plot or not (default =True)save_path: path to where the plot will be saved;Noneimplies the graph will not be saved (default =None)