Package: bwsample
bwsample: Sampling and Evaluation of Best-Worst Scaling setsSampling algorithm for best-worst scaling (BWS) sets, extracting pairs from evaluated BWS sets, count in dictionary of keys sparse matrix, and compute scores based on it.The packagebwsampleaddresses three areas:SamplingCountingRankingInstallationThebwsamplegit repois available asPyPi packagepipinstall"bwsample>=0.7.0"OverviewThebwsamplecan be deployed at different stages to prepare BWS example sets and post-process evaluated BWS sets.
AnActive Learningexperiment using an Web App with BWS user interface to judge sentence examples is shown in the diagram below. Thebwsamplewould be implemented in a (python based) REST API. The App requests new BWS example sets, andbwsample.samplegenerates these. After the App posts the evaluation results to the API,bwsample.countextract new pairwise data from evaluated BWS sets. The pairwise comparision matrix can be used bybwsample.rankto compute  scores for a new updated training set.SamplingInput Data:The input dataexamplesforbwsample.sampleshould be aList[anything].
For example,List[Dict[ID,DATA]]with identifiers using the key"id"and the associated data using the key"data", e.g.examples=[{"id":"id1","data":"data..."},{"id":"id2","data":["other","data"]},{"id":"id3","data":{"key","value"}},{"id":"id4","data":"lorem"},{"id":"id5","data":"ipsum"},{"id":"id6","data":"blind"},{"id":"id7","data":"text"}]Call the function:The number of items per BWS setn_items(M) must be specified, e.g.n_items=4if your App displays four items.
The'overlap'algorithm assigns everyi*(M-1)+1-th example to two consecutive BWS sets, so that1/(M-1)of examples are evaluated two times.
The'twice'algorithm connects the remaining(M-2)/(M-1)non-overlapping from'overlapping'so that all examples occur twice.
The total number of sampled BWS sets might differ accordingly.importbwsampleasbwssamples=bws.sample(examples,n_items=4,method='overlap')Output Data:The output has the following structure[
    [{'id': 'id1', 'data': 'data...'}, {'id': 'id2', 'data': ['other', 'data']}, {'id': 'id3', 'data': {'key', 'value'}}, {'id': 'id4', 'data': 'lorem'}], 
    [{'id': 'id1', 'data': 'data...'}, {'id': 'id4', 'data': 'lorem'}, {'id': 'id5', 'data': 'ipsum'}, {'id': 'id6', 'data': 'blind'}]
]Warning:len(examples)must be a multiple of(n_items - 1)References:Section 5 (page 4) in: Hamster, U. A. (2021, March 9). Extracting Pairwise Comparisons Data from Best-Worst Scaling Surveys by Logical Inference.https://doi.org/10.31219/osf.io/qkxejCountingInput Data:The input dataevaluationsforbwsample.countshould structured asList[Tuple[List[ItemState], List[ItemID]]].
The labelling/annotation application should produce a list of item statesList[ItemState]with the statesBEST:1,WORST:2andNOT:0for each item.
And the corresponding list of IDsList[ItemID]for each item or resp. example.evaluations=(([0,0,2,1],['id1','id2','id3','id4']),([0,1,0,2],['id4','id5','id6','id7']),([1,2,0,0],['id7','id8','id9','id1']))Call the function:importbwsampleasbwsagg_dok,direct_dok,direct_detail,logical_dok,logical_detail=bws.count(evaluations)Output Data:The functionbwsample.countoutputs Dictionary of Keys (DOK) with the data structureDict[Tuple[ItemID, ItemID], int], e.g.agg_dok,direct_dok,direct_detail["bw"], etc., what contain variants which pairs where counted:agg_dokdirect_dokdirect_detail["bw"]--BEST>WORSTdirect_detail["bn"]--BEST>NOTdirect_detail["nw"]--NOT>WORSTlogical_doklogical_detail["nn"]--D>E>F vs X>E>Zlogical_detail["nb"]--D>E>F vs E>Y>Zlogical_detail["nw"]--D>E>F vs X>Y>Elogical_detail["bn"]--D>E>F vs X>D>Zlogical_detail["bw"]--D>E>F vs X>Y>Dlogical_detail["wn"]--D>E>F vs X>F>Zlogical_detail["wb"]--D>E>F vs F>Y>ZLimit the Database Size:Logical Inference has quadratic complexity, and it might be beneficial to limit the database what can be done by thelogical_databaseparameter.importbwsampleasbwsagg_dok,direct_dok,direct_detail,logical_dok,logical_detail=bws.count(evaluations,logical_database=evaluations[:1])Update Frequencies:The functionbwsample.countis an update function, i.e. you can provide previous count or resp. frequency data (e.g.logical_dok) or start from scratch (e.g.agg_dok=None).importbwsampleasbwsevaluations=[...]direct_dok={...}direct_detail={...}logical_dok={...}logical_detail={...}database=[...]agg_dok,direct_dok,direct_detail,logical_dok,logical_detail=bws.count(evaluations,direct_dok=direct_dok,direct_detail=direct_detail,logical_dok=logical_dok,logical_detail=logical_detail,logical_database=database)References:Section 3-4 in: Hamster, U. A. (2021, March 9). Extracting Pairwise Comparisons Data from Best-Worst Scaling Surveys by Logical Inference.https://doi.org/10.31219/osf.io/qkxejRankingInput Data:The input data is a Dictionary of Keys (DoK) object produced bybwsample.count.Call the function:The functionbwsample.rankcomputes a python index variable with a proposed ordering (ranked), and ordered list of example IDs (ordids), scores (scores) and further information depending on the selectedmethod.importbwsampleasbwsranked,ordids,metrics,scores,info=bws.rank(dok,method='ratio',adjust='quantile')Available methods:Computed from extracted pairs:'ratio'-- Simple ratios for each pair, and sum ratios for each item.'approx'-- Chi-Squared based p-value (Hoaglin Approximation) for each pair, and sum 1-pval for each item (Beh et al, 2018)'btl'-- Bradley-Terry-Luce (BTL) model estimated with MM algorithm (Hunter, 2004).'eigen'-- Eigenvectors of the reciprocal pairwise comparison matrix (Saaty, 2003).'trans'-- Estimate transition probability of the next item to be better.The implementationsratio,pvalue,'btl','eigen', and'trans'are fully based on sparse matrix operations andscipy.sparsealgorithms, and avoid accidental conversions to dense matrices.References:Hoaglin Approximation for p-values: Beh, E., 2018. Exploring How to Simply Approximate the P-value of a Chi-squared Statistic. AJS 47, 63–75.https://doi.org/10.17713/ajs.v47i3.757Eigenvector solution in: Saaty, T. L. (2003). Decision-making with the AHP: Why is the principal eigenvector nec- essary. European Journal of Operational Research, 145(1), 85–91.https://doi.org/10.1016/S0377-2217(02)00227-8Estimating the BTL model in: Hunter, D. R. (2004). MM algorithms for generalized Bradley-Terry models. The Annals of Statistics, 32(1), 384–406.https://doi.org/10.1214/aos/1079120141MaxDiff score in: Orme, B. (2009). MaxDiff Analysis: Simple Counting, Individual-Level Logit, and HB.https://sawtoothsoftware.com/uploads/sawtoothsoftware/originals/f89a6537-1cae-4fb5-afad-9d325c2a3143.pdfHamster, U. A. (2021, April 1). Pairwise comparison based ranking and scoring algorithms.https://doi.org/10.31219/osf.io/ev7fwAppendixInstall a virtual environmentIn order to run the Jupyter notebooks or want to work on this project (e.g. unit tests, syntax checks) you should install a Python virtual environment.python3.7-mvenv.venvsource.venv/bin/activate
pipinstall--upgradepip
pipinstall-rrequirements.txt--no-cache-dir
pipinstall-rrequirements-dev.txt--no-cache-dir
pipinstall-rrequirements-demo.txt--no-cache-dir(If your git repo is stored in a folder with whitespaces, then don't use the subfolder.venv. Use an absolute path without whitespaces.)Python commandsJupyter for the examples:jupyter labCheck syntax:flake8 --ignore=F401 --exclude=$(grep -v '^#' .gitignore | xargs | sed -e 's/ /,/g')Run Unit Tests:pytestPublish# pandoc README.md --from markdown --to rst -s -o README.rstpythonsetup.pysdisttwinecheckdist/*
twineupload-rpypidist/*Clean upfind.-typef-name"*.pyc"|xargsrm
find.-typed-name"__pycache__"|xargsrm-r
rm-r.pytest_cache
rm-r.venvSupportPleaseopen an issuefor support.ContributingPlease contribute usingGithub Flow. Create a branch, add commits, andopen a pull request.AcknowledgementsThis work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) -433249742. Project duration: 2020-2023.CitationPlease cite the peer-reviewed JOSS paperwhen using this software for any purpose.