Package: dtlpymetrics
Scoring and metrics appDescriptionThis app improves the efficiency of annotating data by improving annotation quality while reducing the time required to
produce them.The components of this app are:Functions to calculate scores for quality tasks and model predictions.Custom nodes that can be added to pipelines to calculate scores when a task's quality items are completed.Also checkthis notebookfor more informationQuality FlowsTo understand more about each of these tasks, refer to the main Dataloop documentation linked:Qualification tasksHoneypot tasksConsensus tasksIn general, an annotator will receive an assignment to complete their annotation task. For a given item in a consensus task,
each assignment will be cross-compared with every other assignment. In the case of qualification and honeypot tasks,
each item will only have one assignment associated with it.What's Supported?Supported file types:imagevideoScoring is currently supported for quality tasks with the following annotation types (with geometry score method in parentheses, where applicable):classificationbounding box (IOU)polygon (IOU)segmentation (IOU)point (distance)Score TypesDuring scoring, the following scores will be created for each annotation:raw_annotation_scores-  for each annotation comparison we havegeometry,labelandattributematching scoresannotation_overall- the mean of each annotation’s raw scoresuser_confusion_score- the mean of every annotation overall score, relative to ref or another assigneeitem_confusion_score- the count of the number of label pairs associated with the assignee’s label, relative to the reference’s labelitem_overall_score- the mean value ofeachannotation overall score associated with an item1) Raw annotation scores:There are three types of scores for annotations:annotation_iou,annotation_labelandannotation_attribute.These scores can be determined by the user, and the default is to include all three scores, and the default value is 1 (which can be modified).2) Annotation overallForannotation_overallscore we calculate the mean value for all raw annotation scores per annotation.3) User confusion scoreTheuser_confusionscore represents the mean annotation score a given assignee has, relative to raw scores when comparing it to another set of annotations (either the reference or another assignee).4) Label confusion scoreThelabel_confusionscore represents the count for a label annotated by a given assignee, relative to label each label class in the other set of annotations (either reference or another assignee).5) Item overall scoreTheitem_overallscore is the mean value of all annotations associated with an item, averaging the mean overall annotation score.Any calculated and uploaded scores will replace any previous scores for all items of a given task.Note about videos: Video scores will differ slightly from image scores. Video sores are calculated frame by frame, and then specific annotation scores will be the average of these scores across all relevant frames for that specific annotation. Confusion scores are not calculated due to the multi-frame nature of videos. Item overall scores remain an average of all annotations of the video item.Confusion ExampleThere are generally two kinds of scores: regular scores, and “confusion” scores.Regular scores show the level of agreement or overlap between two sets of annotations. They use the ID of the entities being compared for theentityIDandrelativefields. This can be for comparing annotations or items.valuewill typically be a number between 0 and 1.There are two types of confusion scores: item label confusion, and user confusion.Item label confusionshows the number of instances in which an assignee’s label corresponds with the ground truth labels.Ground truth annotations:item = dl.items.dl(item_id='64c0fc0730b03f27ca3a58db')Assignee annotations:item = dl.items.dl(item_id='64c0f2e1ec9103d52eaedbe2')In this example item, the ground truth has 3 for each cat and dog class. The assignee however, labels 1 as cat and 5 as dog. This would result in the following item label confusion scores:{"type":"label_confusion","value":1,"entityId":"cat","context":{"relative":"cat","taskId":"<TASK_ID>","itemId":"<ITEM_ID">,"datasetId":"<DATASET_ID>"}},{"type":"label_confusion","value":3,"entityId":"dog","context":{"relative":"dog","taskId":"<TASK_ID>","itemId":"<ITEM_ID">,"datasetId":"<DATASET_ID>"}},{"type":"label_confusion","value":2,"entityId":"dog","context":{"relative":"cat","taskId":"<TASK_ID>","itemId":"<ITEM_ID">,"datasetId":"<DATASET_ID>"}}Python installationpipinstalldtlpymetricsFunctionsSeethis pagefor details on additional functions.Contributions, Bugs and Issues - How to ContributeWe welcome anyone to help us improve this app.Hereare detailed instructions to help you open a bug or ask for a feature request