Package: posemaro
Dataset Management Framework (Datumaro)A framework and CLI tool to build, transform, and analyze datasets.VOC dataset                                  ---> Annotation tool
     +                                     /
COCO dataset -----> Datumaro ---> dataset ------> Model training
     +                                     \
CVAT annotations                             ---> Publication, statistics etc.Table of ContentsExamplesFeaturesInstallationUsageUser manualContributingExamples(Back to top)Convert PASCAL VOC dataset to COCO format, keep only images withcatclass
presented:# Download VOC dataset:# http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tardatumconvert--input-formatvoc--input-path<path/to/voc>\--output-formatcoco\--filter'/item[annotation/label="cat"]'\----reindex1# avoid annotation id conflictsConvert only non-occludedannotations from aCVATproject to TFrecord:# export Datumaro dataset in CVAT UI, extract somewhere, go to the project dirdatumfilter-e'/item/annotation[occluded="False"]'\--modeitems+anno--output-dirnot_occluded
datumexport--projectnot_occluded\--formattf_detection_api----save-imagesAnnotate MS COCO dataset, extract image subset, re-annotate it inCVAT, update old dataset:# Download COCO dataset http://cocodataset.org/#download# Put images to coco/images/ and annotations to coco/annotations/datumimport--formatcoco--input-path<path/to/coco>
datumexport--filter'/image[images_I_dont_like]'--formatcvat\--output-dirreannotation# import dataset and images to CVAT, re-annotate# export Datumaro project, extract to 'reannotation-upd'datummergereannotation-upd
datumexport--formatcocoAnnotate instance polygons inCVAT, export
as masks in COCO:datumconvert--input-formatcvat--input-path<path/to/cvat.xml>\--output-formatcoco----segmentation-modemasksApply an OpenVINO detection model to some COCO-like dataset,
then compare annotations with ground truth and visualize in TensorBoard:datumimport--formatcoco--input-path<path/to/coco># create model results interpretation scriptdatummodeladdmymodelopenvino\--weightsmodel.bin--descriptionmodel.xml\--interpretation-scriptparse_results.py
datummodelrun--modelmymodel--output-dirmymodel_inference/
datumdiffmymodel_inference/--formattensorboard--output-dirdiffChange colors in PASCAL VOC-like.pngmasks:datumimport--formatvoc--input-path<path/to/voc/dataset># Create a color map file with desired colors:## label : color_rgb : parts : actions# cat:0,0,255::# dog:255,0,0::## Save as mycolormap.txtdatumexport--formatvoc_segmentation----label-mapmycolormap.txt# add "--apply-colormap=0" to save grayscale (indexed) masks# check "--help" option for more info# use "datum --loglevel debug" for extra conversion infoCreate a custom COCO-like dataset:importnumpyasnpfromdatumaro.components.extractorimport(DatasetItem,Bbox,LabelCategories,AnnotationType)fromdatumaro.components.datasetimportDatasetdataset=Dataset(categories={AnnotationType.label:LabelCategories.from_iterable(['cat','dog'])})dataset.put(DatasetItem(id=0,image=np.ones((5,5,3)),annotations=[Bbox(1,2,3,4,label=0),]))dataset.export('test_dataset','coco')Features(Back to top)Dataset reading, writing, conversion in any direction.Supported formats:COCO(image_info,instances,person_keypoints,captions,labels,panoptic,stuff)PASCAL VOC(classification,detection,segmentation,action_classification,person_layout)YOLO(bboxes)TF Detection API(bboxes,masks)WIDER Face(bboxes)VGGFace2(landmarks,bboxes)MOT sequencesMOTS PNGImageNetCIFAR-10/100(classification)MNIST(classification)MNIST in CSV(classification)CamVidCityscapesKitti(segmentation,detection,3D raw/velodyne points)Supervisely(point cloud)CVATLabelMeICDAR13/15(word_recognition,text_localization,text_segmentation)Market-1501(person re-identification)LFW(classification,person re-identification,landmarks)Dataset buildingMerging multiple datasets into oneDataset filtering by a custom criteria:remove polygons of a certain classremove images without annotations of a specific classremoveoccludedannotations from imageskeep only vertically-oriented imagesremove small area bounding boxes from annotationsAnnotation conversions, for instance:polygons to instance masks and vice-versaapply a custom colormap for mask annotationsrename or remove dataset labelsSplitting a dataset into multiple subsets liketrain,val, andtest:random splittask-specific splits based on annotations,
which keep initial label and attribute distributionsfor classification task, based on labelsfor detection task, based on bboxesfor re-identification task, based on labels,
avoiding having same IDs in training and test splitsSampling a datasetanalyzes inference result from the given dataset
and selects the ‘best’ and the ‘least amount of’ samples for annotation.Select the sample that best suits model training.sampling with Entropy based algorithmDataset quality checkingSimple checking for errorsComparison with model inferenceMerging and comparison of multiple datasetsAnnotation validation based on the task type(classification, etc)Dataset comparisonDataset statistics (image mean and std, annotation statistics)Model integrationInference (OpenVINO, Caffe, PyTorch, TensorFlow, MxNet, etc.)Explainable AI (RISE algorithm)RISE for classificationRISE for object detectionCheckthe design documentfor a full list of features.
Checkthe user manualfor usage instructions.Installation(Back to top)DependenciesPython (3.6+)Optional: OpenVINO, TensforFlow, PyTorch, MxNet, Caffe, Accuracy CheckerOptionally, create a virtual environment:python-mpipinstallvirtualenv
python-mvirtualenvvenv
.venv/bin/activateInstall Datumaro package:pipinstalldatumaroUsage(Back to top)There are several options available:A standalone command-line toolA python moduleStandalone toolDatuaro as a standalone tool allows to do various dataset operations from
the command line interface:datum--help
python-mdatumaro--helpPython moduleDatumaro can be used in custom scripts as a Python module. Used this way, it
allows to use its features from an existing codebase, enabling dataset
reading, exporting and iteration capabilities, simplifying integration of custom
formats and providing high performance operations:fromdatumaro.components.projectimportProject# load a Datumaro projectproject=Project.load('directory')# create a datasetdataset=project.make_dataset()# keep only annotated imagesdataset.select(lambdaitem:len(item.annotations)!=0)# change dataset labelsdataset.transform('remap_labels',{'cat':'dog',# rename cat to dog'truck':'car',# rename truck to car'person':'',# remove this label},default='delete')# remove everything else# iterate over dataset elementsforitemindataset:print(item.id,item.annotations)# export the resulting dataset in COCO formatdataset.export('dst/dir','coco')Check ourdeveloper guidefor additional
information.Contributing(Back to top)Feel free toopen an Issue, if you
think something needs to be changed. You are welcome to participate in
development, instructions are available in ourcontribution guide.