Package: data-gradients
DataGradientsDataGradients is an open-source python based library designed forcomputer vision dataset analysis.Extractvaluable insightsfrom your datasets and getcomprehensive reports effortlessly.🔍 Detect Common Data IssuesCorrupted dataLabeling errorsUnderlying biases, and more.💡 Extract Insights for Better Model DesignInformed decisions based on data characteristics.Object size and location distributions.High frequency details.🎯 Reduce Guesswork for HyperparametersDefine the correct NMS and filtering parameters.Identify class distribution issues.Calibrate metrics for your unique dataset.🛠 CapabilitiesNon-exhaustive list of supported features.General Image Metrics: Explore key attributes like resolution, color distribution, and average brightness.Class Overview: Get a snapshot of class distributions, most frequent classes, and unlabelled images.Positional Heatmaps: Visualize where objects tend to appear within your images.Bounding Box & Mask Details: Delve into dimensions, area coverages, and resolutions of objects.Class Frequencies Deep Dive: Dive deeper into class distributions, understanding anomalies and rare classes.Detailed Object Counts: Examine the granularity of components per image, identifying patterns and outliers.Andmany more!📘Deep Dive into Data ProfilingPuzzled by some dataset challenges while using DataGradients? We've got you covered.Enrich your understanding with this🎓free online course. Dive into dataset profiling, confront its complexities, and harness the full potential of DataGradients.Example of pages from the ReportExample of specific featuresCheck out thepre-computed dataset analysisfor a deeper dive into reports.Table of ContentsInstallationQuick StartPrerequisitesDataset AnalysisReportFeature ConfigurationDataset ExtractorsPre-computed Dataset AnalysisLicenseInstallationYou can install DataGradients directly from the GitHub repository.pip install data-gradientsQuick StartPrerequisitesDataset: Includes aTrainset and aValidationor aTestset.Dataset Iterable: A method to iterate over your Dataset providing images and labels. Can be any of the following:PyTorchDataloaderPyTorchDatasetGenerator that yields image/label pairsAny other iterable you use for model training/validationOne of:Class Names: Either the list of all class names in the dataset OR dictionary mapping ofclass_id->class_name.Number of classes:  Indicate how many unique classes are in your dataset. Ensure this number is greater than the highest class index (e.g., if your highest class index is 9, the number of classes should be at least 10).Please ensure all the points above are checked before you proceed withDataGradients.Examplefromtorchvision.datasetsimportCocoDetectiontrain_data=CocoDetection(...)val_data=CocoDetection(...)class_names=["person","bicycle","car","motorcycle",...]# OR# class_names = {0: "person", 1:"bicycle", 2:"car", 3: "motorcycle", ...}Good to Know- DataGradients will try to find out how the dataset returns images and labels.If something cannot be automatically determined, you will be asked to provide some extra information through a text input.In some extreme cases, the process will crash and invite you to implement a customdataset extractorHeads up- DataGradients provides a few out-of-the-boxdataset/dataloaderimplementation.
You can find more dataset implementations inPyTorchorSuperGradients.Dataset AnalysisYou are now ready to go, chose the relevant analyzer for your task and run it over your datasets!Image Classificationfromdata_gradients.managers.classification_managerimportClassificationAnalysisManagertrain_data=...# Your dataset iterable (torch dataset/dataloader/...)val_data=...# Your dataset iterable (torch dataset/dataloader/...)class_names=...# [<class-1>, <class-2>, ...]analyzer=ClassificationAnalysisManager(report_title="Testing Data-Gradients Classification",train_data=train_data,val_data=val_data,class_names=class_names,)analyzer.run()Object Detectionfromdata_gradients.managers.detection_managerimportDetectionAnalysisManagertrain_data=...# Your dataset iterable (torch dataset/dataloader/...)val_data=...# Your dataset iterable (torch dataset/dataloader/...)class_names=...# [<class-1>, <class-2>, ...]analyzer=DetectionAnalysisManager(report_title="Testing Data-Gradients Object Detection",train_data=train_data,val_data=val_data,class_names=class_names,)analyzer.run()Semantic Segmentationfromdata_gradients.managers.segmentation_managerimportSegmentationAnalysisManagertrain_data=...# Your dataset iterable (torch dataset/dataloader/...)val_data=...# Your dataset iterable (torch dataset/dataloader/...)class_names=...# [<class-1>, <class-2>, ...]analyzer=SegmentationAnalysisManager(report_title="Testing Data-Gradients Segmentation",train_data=train_data,val_data=val_data,class_names=class_names,)analyzer.run()ExampleYou can test the segmentation analysis tool in the followingexamplewhich does not require you to download any additional data.ReportOnce the analysis is done, the path to your pdf report will be printed. You can find here examples ofpre-computed dataset analysis reports.Feature ConfigurationThe feature configuration allows you to run the analysis on a subset of features or adjust the parameters of existing features.
If you are interested in customizing this configuration, you can check out thedocumentationon that topic.Dataset ExtractorsEnsuring Comprehensive Dataset CompatibilityDataGradients is adept at automatic dataset inference; however, certain specificities, such as nested annotations structures or unique annotation format, may necessitate a tailored approach.To address this, DataGradients offersextractorstailored for enhancing compatibility with diverse dataset formats.For an in-depth understanding and implementation details, we encourage a thorough review of theDataset Extractors Documentation.Pre-computed Dataset AnalysisExample notebook on ColabDetectionCommon DatasetsCOCOVOCRoboflow 100Datasets4-fold-defectabdomen-mriacl-x-rayactivity-diagrams-qdobraerial-cowsaerial-poolaerial-spheresanimals-ij5d2apex-videogameapples-fvpl5aquarium-qlnqyasbestosavatar-recognition-nuexeaxial-mribacteria-ptywibccd-ouzjzbees-jt5inbone-fracture-7fylgbrain-tumor-m2pbpcable-damagecables-nl42kcavity-rs0ufcell-towerscells-uyemfchess-pieces-mjzgjcircuit-elementscircuit-voltagescloud-typescoins-1apkiconstruction-safety-gsnvbcoral-lwptlcorrosion-bi3q3cotton-20xz5cotton-plant-diseasecsgo-videogamecurrency-v4f8jdigits-t2eg6document-partsexcavators-czvg9farcry6-videogamefish-market-ggjsoflir-camera-objectsfurniture-ngpeagauge-u2lwvgrass-weedsgynecology-mrihalo-infinite-angel-videogamehand-gestures-jps7zinsects-mytwuleaf-disease-nsdsrlettuce-palletsliver-diseasemarblesmask-wearing-608prmitosis-gjs3gnumber-opspaper-partsparagraphs-co84bparasites-1s07hpeanuts-sd4kfpeixos-fishpeople-in-paintingspests-2xlvxphagespills-sxdhtpoker-cards-cxcvzprinted-circuit-boardradio-signalroad-signs-6ih4yroad-trafficrobomasters-285kmsecondary-chainssedimentary-features-9eosfshark-teeth-5atkusign-language-sokdrsignatures-xc8upsmoke-uvyljsoccer-players-5fuqssoda-bottlessolar-panels-taxvbstomata-cellsstreet-worktabular-data-wf9uhteam-fight-tacticsthermal-cheetah-my4dpthermal-dogs-and-people-x6ejwtrail-cameratruck-movementtweeter-poststweeter-profileunderwater-objects-5v7p8underwater-pipes-4ng4tuno-deckvalentines-chocolatevehicles-q0x2vwall-damagewashroom-rf1faweed-crop-aerialwine-labelsx-ray-rheumatologySegmentationCOCOCityspaceVOCCommunityClick here to join our Discord CommunityLicenseThis project is released under theApache 2.0 license.