Package: py-irt
py-irtBayesian IRT models in PythonOverviewThis repository includes code for fitting Item Response Theory (IRT) models using variational inference.At present, the one parameter logistic (1PL) model, aka Rasch model, two parameter logistic model (2PL) and four parameter logistic model (4PL) have been implemented.
The user can specify whether vague or hierarchical priors are used.
The three-parameter logistic model is in the pipeline and will be added when available.Licensepy-irt is licensed under theMIT license.Installationpy-irt is now available on PyPi!Pre-reqsInstallPyTorch.InstallPyroInstall py-irt:pipinstallpy-irtORInstallPoetrygitclonehttps://github.com/nd-ball/py-irt.gitcdpy-irt
poetryinstallUsageOnce you install from PyPI, you can use the following command to fit an IRT
model on the scored predictions of a dataset. For example, if you were to run py-irt with the 4PL model on the scored predictions of different transformer models on the SQuAD dataset, you'd do this:py-irt train 4pl ~/path/to/dataset/eg/squad.jsonlines /path/to/output/eg/test-4pl/FAQWhat kind of output should I expect on running the command to train an IRT model?You should see something like this when you run the command given above:I tried installing py-irt using pip from PyPI. But when I try to run the commandpy-irt train 4pl ~/path/to/dataset/eg/squad.jsonlines /path/to/output/eg/test-4pl/, I get an error that saysbash: py-irt: command not found. How do I fix this?The CLI interface was implemented in PyPi version 0.2.1. If you are getting this error try updating py-irt:pip install --upgrade py-irtAlternatively, you can install the latest version from github:gitclonehttps://github.com/nd-ball/py-irt.gitcdpy-irt
mv~/py-irt/py_irt/cli.py~/py-irt/
pythoncli.pytrain4pl~/path/to/dataset/eg/squad.jsonlines/path/to/output/eg/test-4pl/How do I evaluate a trained IRT model?If you have already trained an IRT model you can use the following command:py-irt evaluate 4pl ~/path/to/data/best_parameters.json ~/path/to/data/test_pairs.jsonlines /path/to/output/eg/test-4pl/Wheretest_pairs.jsonlinesis a jsonlines file with the following format:{"subject_id": "ken", "item_id": "q1"}
{"subject_id": "ken", "item_id": "q2"}
{"subject_id": "burt", "item_id": "q1"}
{"subject_id": "burt", "item_id": "q3"}If you would like to both train and evaluate a model you can use the following command:py-irt train-and-evaluate 4pl ~/path/to/data/squad.jsonlines /path/to/output/eg/test-4pl/By default this will train a model with 90% of the provided data and evaluate with the remaining 10%.
To change this behavior you can add--evaluation allto the command above.
The model will train and evaluate against all of the data.CitationsIf you use this code, please consider citing the following papers:@inproceedings{lalor2019emnlp,author={Lalor,JohnPandWu,HaoandYu,Hong},title={LearningLatentParameterswithoutHumanResponsePatterns:ItemResponseTheorywithArtificialCrowds},year={2019},booktitle={Proceedingsofthe2019ConferenceonEmpiricalMethodsinNaturalLanguageProcessing},}@inproceedings{rodriguez2021evaluation,title={EvaluationExamplesAreNotEquallyInformative:HowShouldThatChangeNLPLeaderboards?},author={Rodriguez,PedroandBarrow,JoeandHoyle,AlexanderMiserlisandLalor,JohnPandJia,RobinandBoyd-Graber,Jordan},booktitle={Proceedingsofthe59thAnnualMeetingoftheAssociationforComputationalLinguisticsandthe11thInternationalJointConferenceonNaturalLanguageProcessing(Volume1:LongPapers)},pages={4486--4503},year={2021}}Implementation is based on the following paper:@article{natesan2016bayesian,title={BayesianpriorchoiceinIRTestimationusingMCMCandvariationalBayes},author={Natesan,PrathibaandNandakumar,RatnaandMinka,TomandRubright,JonathanD},journal={Frontiersinpsychology},volume={7},pages={1422},year={2016},publisher={Frontiers}}ContributingThis is research code. Pull requests and issues are welcome!Questions?Let me know if you have any requests, bugs, etc.Email:john.lalor@nd.edu