Package: ml4qc
Theml4qcPython package offers a toolkit for employing machine learning technologies
in survey data quality control. Among other things, it helps to extendthe surveydata packageand advanceSurveyCTO’s
machine learning roadmap.
See further below for more about the research program and specifics on the package.InstallationInstalling the latest version with pip:pip install ml4qcOverview of research programThe working title and abstract for the overarching research program is as follows:Can machine learning aid survey data quality-control efforts, even without access to actual
survey data?A robust quality-control process with some degree of human review is often crucial for survey
data quality, but resources for human review are almost always limited and therefore rationed.
While traditional statistical methods of directing quality-control efforts often rely on
field-by-field analysis to check for outliers, enumerator effects, and unexpected patterns,
newer machine-learning-based methods allow for a more holistic evaluation of interviews. ML
methods also allow for human review to train models that can then predict the results of
review, increasingly focusing review time on potential problems. In this research program, we
explore the potential of ML-based methods to direct and supplement QC efforts across
international settings. In particular, we look at the potential for privacy-protecting
approaches that allow ML models to be trained and utilized without ever exposing
personally-identifiable data — or, indeed, any survey data at all — to ML systems or analysts.
Specifically, metadata and paradata, including rich but non-identifying data from mobile device
sensors, is used in lieu of potentially-sensitive survey data.We currently envision three phases to the work:Phase 1: Foundation-building and early explorationGoal 1: Establish a flexible codebase to serve as a platform for exploration and analysisGoal 2: Test whether popular supervised prediction models work to predict quality classifications when trained
on meta/paradataGoal 3: Test whether popular unsupervised models can identify useful patterns in enumerator effectsPhase 2: Dig deeper across settings, refine tools and understandingGoal 1: Tune models and features for supervised models, establish links between review processes, quality
classifications, and effectivenessGoal 2: Find useful structure for measuring and reporting enumerator effects, how to control out nonrandom
sources of variationGoal 3: For all models, establish requirements for training data (e.g., sample sizes)Phase 3: Test the potential for transfer learning, further refine toolsGoal 1: Develop instrument-agnostic meta/paradata format and test the potential for transfer learning to
enable useful results earlier in the launch of a new surveyGoal 2: Support continued scaling of experimentation across more settings with easy, production-ready toolsWe are currently in Phase 1. See the “Examples” section below for early results.Overview of Python packageTheml4qcpackage builds on thescikit-learntoolset. It includes the following
utility classes for working with survey data:SurveyMLprovides core functionality, including preprocessing, outlier detection, and cluster analysisSurveyMLClassifierbuilds onSurveyML, adding support for running classification models and reporting out
resultsWhileSurveyMLClassifiersupports a variety of approaches, the currently-recommended
approach to binary classification is as follows:Donotreweight for class imbalances; useSurveyMLClassifier.cv_for_best_hyperparameters()to find the optimal hyperparameters
for a given dataset, withneg_log_loss,neg_brier_score, orroc_aucas the CV metric
to optimize. This will optimize for an unbiased distribution of estimated probabilities.Use acalibration_method(isotonicorsigmoid) to calibrate the estimated
probability distribution.Almost always (and especially when classes are imbalanced), specify a non-default option
for the classificationthreshold(and possiblythreshold_value), as the
default threshold of 0.5 is unlikely to be optimal. When in doubt, usethreshold='optimal_f'to choose the threshold that maximizes the F-1 score.This is essentially the approach used in the examples linked below.When there are nonrandom aspects to interview assignment, it is also recommended to initializeSurveyMLClassifierwith a list ofcontrol_featuresto control out as much of the nonrandom-assignment effects as possible. During
pre-processing,control_featureswill be used in OLS regressions to predict each other feature value, and it will
be the residuals that are used in further analysis. This can be particularly important in distinguishing enumerator
effects from the effects of nonrandom assignment. See theCAPI2example linked below.ExamplesThis package is best illustrated by way of example. The following example analyses are available:CATI1 analysisSetting: CATI survey in AfghanistanReview and quality classification strategy: Holistic review of all submissionsSupervised results: Precision for predicting rejected submission ranges from 10% to 43% (against a base rate of
3.8%)Unsupervised results: Significant enumerator effects discovered and summarizedCATI2 analysisSetting: CATI survey in AfghanistanReview and quality classification strategy: Holistic review of all submissionsSupervised results: Precision for predicting rejected submission ranges from 20-50% (against a base rate of 4.7%),
but wide variation due to very small training sampleUnsupervised results: Significant enumerator effects discovered and summarized, but not at cluster levelCATI3 analysisSetting: CATI survey in AfghanistanReview and quality classification strategy: All completed interviews classified the same, all incomplete
interviews rejectedSupervised results: Because there are clear meta/paradata differences between complete and incomplete interviews,
all ML models achieve 100% precision in predicting rejectionUnsupervised results: Very significant enumerator effects discovered and summarizedCAPI1 analysisSetting: CAPI survey in AfghanistanReview and quality classification strategy: Holistic review of all submissionsSupervised results: With only 5 rejected submissions, instead sought to predict “not approved as GOOD quality”
with a base rate of 70% (resting almost completely on the distinction between a “GOOD” and an “OKAY” quality
rating); none of the models succeed in predicting the distinction and it’s not clear that a larger sample size
would helpUnsupervised results: Very significant enumerator effects discovered and summarizedCAPI2 analysisSetting: CAPI survey in EthiopiaReview and quality classification strategy: Submissions flagged with automated statistical checks at the question
level, plus randomly-selected interviews, reviewed for individual responses in need of correction; those that
require correction classified as “OKAY” (vs. “GOOD”) qualitySupervised results: Full results still TBD, but predictive results poor overall, though slightly better with
structural models (logistic regression and neural networks)Unsupervised results: Once many of the effects of nonrandom assignment are controlled out, there are only
weak enumerator effects at the cluster levelDocumentationSee the full reference documentation here:https://ml4qc.readthedocs.io/Project supportDobilityhas generously provided financial and other support for v1 of theml4qcpackage, including support for early testing and piloting.DevelopmentTo develop locally:git clonehttps://github.com/orangechairlabs/ml4qc.gitcd ml4qcpython-mvenv venvsource venv/bin/activatepip install-rrequirements.txtFor convenience, the repo includes.ideaproject files for PyCharm.To rebuild the documentation:Update version number in/docs/source/conf.pyUpdate layout or options as needed in/docs/source/index.rstIn a terminal window, from the project directory:cd docsSPHINX_APIDOC_OPTIONS=members,show-inheritancesphinx-apidoc-osource../src/ml4qc--separate--forcemake clean htmlTo rebuild the distribution packages:For the PyPI package:Update version number (and any build options) in/setup.pyConfirm credentials and settings in~/.pypircRun/setup.pyforbdist_wheelbuild type (Tools… Run setup.py task…in PyCharm)Delete old builds from/distIn a terminal window:twine upload dist/*--verboseFor GitHub:Commit everything to GitHub and merge tomainbranchAdd new release, linking to new tag likev#.#.#in main branchFor readthedocs.io:Go tohttps://readthedocs.org/projects/ml4qc/, log in, and click to rebuild from GitHub (only if it doesn’t automatically trigger)