Package: pyretest
PyretestLibrary to measure test-retest reliability and to estimate adequate sample size using simulated questionnaire responses.The library offers an unweighted and weighted of the pooled Cohen's Kappa.A sampler is also provided to generate a sample of questionnaire responses.References:Cohen, J. (1960). A coefficient of determination for the case of nominal scales. Educational and Psychological Measurement, 20(1), 37-46.De Vries, H., Elliott, M. N., Kanouse, D. E., & Teleki, S. S. (2008). Using pooled kappa to summarize interrater agreement across many items. Field methods, 20(3), 272-282.Installationpip install pyretestUsageComputing the pooled Cohen's KappaComputing the pooled Cohen's Kappa assuming a reliability of 10%.# Import the librariesimportnumpyasnpfrompyretestimportpooled_cohen_kappa,sample_questionnaire,Question# Define a simple questionnaire with 5 questions, each with 4 answersquestions=[Question(["a","b","c","d"],np.random.rand(4)),Question(["a","b","c","d"],np.random.rand(4)),Question(["a","b","c","d"],np.random.rand(4)),Question(["a","b","c","d"],np.random.rand(4)),]# Sample 1000 questionnaire responses twice (e.g. two raters, or two endpoints)samples_a=np.array(sample_questionnaire(questions,n=1000))samples_b=np.array(sample_questionnaire(questions,n=1000))# Set the reliability to 10%# You can also use the function#   make_reliable(samples_a, samples_b, reliability=0.1)reliability=0.1n_reliable=int(reliability*samples_a.shape[0])samples_a[:n_reliable]=samples_b[:n_reliable]# Compute the pooled Cohen's Kappak1=pooled_cohen_kappa(samples_a,samples_b)assertabs(k1-reliability)<0.01Estimate the sample size using bootstrapping# Import the librariesimportnumpyasnpfrompyretestimportQuestion,bootstrap_sample_size_cohen_kappaquestions=[Question(["a","b","c","d","e"],[1/5]*5),Question(["a","b","c","d","e"],[1/5]*5),Question(["a","b","c","d","e"],[1/5]*5),Question(["a","b","c","d","e"],[1/5]*5),]# Define power and type I error ratebeta=0.8alpha=0.05# Assume a reliability of 10%reliability=0.1# Define the number of bootstrap iterationsn_bootstrap=1000# Define the range of sample sizes to teststart_n=10max_n=100n_step=10# Compute the sample sizeresults=bootstrap_sample_size_cohen_kappa(questions,max_n=max_n,weight_type=None,start_n=start_n,n_step=n_step,reliability=reliability,n_bootstrap=n_bootstrap,alpha=alpha,beta=beta)print('Sample size:',results.sample_size)print('Intermediate results df:',results.df)Use weighted versionsTo use the weighted versions of the previous functions, you need to provide aweight_typeargument which can either be"linear"or"quadratic". Seethese slidesfor more details.
You also need to provide a list of Questions.For example:# Assuming code from the previous examplesweight_type="linear"k1_weighted=pooled_cohen_kappa(samples_a,samples_b,weight_type=weight_type,questions=questions)weight_type="quadratic"results=bootstrap_sample_size_cohen_kappa(questions,max_n=max_n,weight_type=weight_type,start_n=start_n,n_step=n_step,reliability=reliability,n_bootstrap=n_bootstrap,alpha=alpha,beta=beta)NoteThere is aseedparameter in the previous functions which can be used to get reproducible samples.If you usesample_questionnaireto sample manually, do not pass the seed twice or you will get the same results for the samples.You can set the seed yourself, with:importrandomrandom.seed(seed)Or set it only in the first call tosample_questionnaire.AuthorsAlbert BuchardMIT License