Package: doubtlab
doubtlabA lab for bad labels. Learn morehere.This repository contains general tricks that may help you find bad, or noisy, labels in your dataset. The hope is that this repository makes it easier for folks to quickly check their own datasets before they invest too much time and compute on gridsearch.InstallYou can install the tool viapiporconda.Install with pippython -m pip install doubtlabInstall with condaconda install -c conda-forge doubtlabQuickstartDoubtlab allows you to define "reasons" for a row of data to deserve another look. These reasons can form a pipeline which can be used to retreive a sorted list of examples worth checking again.fromsklearn.datasetsimportload_irisfromsklearn.linear_modelimportLogisticRegressionfromdoubtlab.ensembleimportDoubtEnsemblefromdoubtlab.reasonimportProbaReason,WrongPredictionReason# Let's say we have some dataset/model alreadyX,y=load_iris(return_X_y=True)model=LogisticRegression(max_iter=1_000)model.fit(X,y)# Next we can add reasons for doubt. In this case we're saying# that examples deserve another look if the associated proba values# are low or if the model output doesn't match the associated label.reasons={'proba':ProbaReason(model=model),'wrong_pred':WrongPredictionReason(model=model)}# Pass these reasons to a doubtlab instance.doubt=DoubtEnsemble(**reasons)# Get the ordered indices of examples worth checking againindices=doubt.get_indices(X,y)# Get dataframe with "reason"-ing behind the sortingpredicates=doubt.get_predicates(X,y)FeaturesThe library implemented many "reasons" for doubt.General ReasonsRandomReason: assign doubt randomly, just for sureOutlierReason: assign doubt when the model declares a row an outlierClassification ReasonsProbaReason: assign doubt when a models' confidence-values are low for any labelWrongPredictionReason: assign doubt when a model cannot predict the listed labelShortConfidenceReason: assign doubt when the correct label gains too little confidenceLongConfidenceReason: assign doubt when a wrong label gains too much confidenceDisagreeReason: assign doubt when two models disagree on a predictionCleanlabReason: assign doubt according tocleanlabMarginConfidenceReason: assign doubt when there's a small difference between the top two class confidencesRegression ReasonsAbsoluteDifferenceReason: assign doubt when the absolute difference is too highRelativeDifferenceReason: assign doubt when the relative difference is too highStandardizedErrorReason: assign doubt when the absolute standardized residual is too highFeedbackIt is early days for the project. The project should be plenty useful as-is, but we
prefer to be honest. Feedback and anecdotes are very welcome!Related ProjectsThecleanlabproject was an inspiration for this one. They have a great heuristic for bad label detection but I wanted to have a library that implements many. Be sure to check out their work on thelabelerrors.comproject.My former employer,Rasa, has always had a focus on data quality. Some of that attitude is bound to have seeped in here. Be sure to check theConversation Driven Developmentapproach andRasa Xif you're working on virtual assistants.My current employer,Explosion, has a neat labelling tool calledProdigy. I'm currently investigating how tools like doubtlab might lead to better labels when combined with this (very like-able) annotation tool.