Package: vl-datasets
VL-DatasetsOpen, Clean, Curated Datasets for Computer Vision🔥 We usefastdup- a free tool to clean all datasets shared in this repo.Explore the docs »Report Issues·Read Blog·Get In Touch·About UsDescriptionvl-datasetsis a Python package that provides access to clean computer vision datasets with only 2 lines of code.For example, to get access to the clean version of theFood-101dataset simply run:We support some of the most widely used computer vision datasets.Let us knowif you have additional request to support a new dataset.All the datasets are analyzed for issues such as:Duplicates.Near Duplicates.Broken images.Outliers.Dark/Bright/Blurry images.Mislabels.Data Leakage.Why?Computer vision is an exciting and rapidly advancing field, with new techniques and models emerging now and then.
However, to develop and evaluate these models, it's essential to have reliable and standardized datasets to work with.Even with the recent success of generative models, data quality remains an issue that'smainly overlooked.
Training models will erroneours data impacts model accuracy, incurs costs in time, storage and computational resources.We believe that access to clean and high-quality computer vision datasets leads to accurate, non-biased, and efficient model.
By providing public access tovl-datasetswe hope it helps advance the field of computer vision.Datasets & Accessvl-datasetsprovides a convenient way to access the cleaned version of the datasets in Python.Alternatively, for each dataset in this repo, we provide a.csvfile that lists the problematic images from the dataset.You can use the listed images in the.csvto improve the model by re-labeling the them or just simply remove it from the dataset.We're a startup and we'd like to offer free access to the datasets as much as we can afford to. But in doing so, we'd also need your support.We're offering select.csvfiles completely free with no strings attached.
For access to our complete dataset and exclusive beta features, all we ask is that yousign upto be a beta tester – it's completely free and your feedback will help shape the future of our platform.Here is a table of widely used computer vision datasets, issues we found and a link to access the.csvfile.DatasetIssuesCSVImport StatementFood-101Duplicates - 0.233 % (235)Outliers - 0.076 % (77)Blur - Blur - 0.183 % (185)Dark - 0.043 % (43)Total- 0.535 % (540)More »Downloadhere.from vl_datasets import VLFood101Oxford-IIIT PetDuplicates - 1.021% (75)Outliers - 0.095% (7)Dark - 0.054% (4)Total- 1.170 % (86)More »Downloadhere.from vl_datasets import VLOxfordIIITPetLAION-1BDuplicates - WIP % (WIP)Outliers - WIP % (WIP)Broken - WIP % (WIP)Blur - WIP % (WIP)Dark - WIP % (WIP)Bright - WIP % (WIP)More »Request accesshere.WIPImageNet-21KDuplicates - 11.853 % (1,559,120)Outliers - 0.085 % (11,119)Blur - 0.292 % (38,458)Dark - 0.179 % (23,574)Bright - 0.431 % (56,754)Mislabels - 3.064 % (402,963)Total- 15.904 % (2,091,988)More »Request accesshere.WIPImageNet-1KDuplicates - 0.520 % (6,660)Outliers - 0.090 % (1,150)Blur - 0.200 % (2,554)Dark - 0.244 % (2,997)Bright - 0.058 % (746)Mislabels - 0.119 % (1,518)Total- 1.221 % (15,625)More »Request accesshere.WIPKITTIDuplicates - 15.294 % (2294)Outliers - 0.107 % (16)Total- 15.401 % (2310)More »Request accesshere.WIPDeepFashionDuplicates - 5.114 % (14,772)Outliers - 0.037 % (107)Total- 5.151 % (14,879)More »Request accesshere.WIPCelebA-HQDuplicates - 1.673 % (3,389)Outliers - 0.077 % (157)Blur - 0.512 % (1,037)Dark - 0.009 % (18)Mislabels - 0.006 % (13)Total- 2.277 % (4,614)More »Request accesshere.WIPCOCODuplicates - 0.123 % (201)Outliers - 0.087 % (143)Blur - 0.029 % (47)Dark - 0.106 % (174)Bright - 0.013 % (21)Total- 0.358 % (586)More »Request accesshere.WIPLearn more on how we clean the datasets using our profilling toolhere.InstallationOption 1- Installvl_datasetspackage fromPyPI:pipinstallvl-datasetsOption 2- Install the bleeding edge version on GitHub:pip install git+https://github.com/visual-layer/vl-datasets.git@main --upgradeUsageTo start usingvl-datasets, import the clean version of the dataset with:fromvl_datasetsimportVLFood101This should import the clean version of theFood101dataset.Next, you can load the dataset as a PyTorchDataset.train_dataset=VLFood101('./',split='train')valid_dataset=VLFood101('./',split='test')If you have a custom.csvfile you can optionally pass in the file:train_dataset=VLFood101('./',split='train',exclude_csv='my-file.csv')The filenames listed in the.csvwill be excluded in the dataset.Next, you can load the train and validation datasets in a PyTorch training loop.See theLearn from Examplessection to learn more.NOTE: Sign upherefor free to be our beta testers and get full access to the all the.csvfiles for the dataset listed in this repo.With the dataset loaded you can train a model using PyTorch training loop.Learn from ExamplesDataset:VLFood101Framework:PyTorch.Description:Load a dataset and train a PyTorch model.Dataset:VLOxfordIIITPetFramework:fast.ai.Description:Finetune a pretrained TIMM model using fastai.Licensevl-datasetsis licensed under the Apache 2.0 License. SeeLICENSE.However, you are bound to the usage license of the original dataset. It is your responsibility to determine whether you have permission to use the dataset under the dataset's license. We provide no warranty or guarantee of accuracy or completeness.Usage TrackingThis repository incorporates usage tracking usingSentry.ioto monitor and collect valuable information about the usage of the application.Usage tracking allows us to gain insights into how the application is being used in real-world scenarios. It provides us with valuable information that helps in understanding user behavior, identifying potential issues, and making informed decisions to improve the application.We DO NOT collect folder names, user names, image names, image content and other personaly identifiable information.What data is tracked?Errors and Exceptions: Sentry captures errors and exceptions that occur in the application, providing detailed stack traces and relevant information to help diagnose and fix issues.Performance Metrics: Sentry collects performance metrics, such as response times, latency, and resource usage, enabling us to monitor and optimize the application's performance.To opt out, define an environment variable namedSENTRY_OPT_OUT.On Linux run the following:exportSENTRY_OPT_OUT=TrueRead more on Sentry's officialwebpage.Getting HelpGet help from the Visual Layer team or community members via the following channels -Slack.GitHubissues.Discussionforum.About Visual-LayerVisual Layer is founded by the authors ofXGBoost,Apache TVM&Turi Create-Danny Bickson,Carlos GuestrinandAmir Alush.Learn more about Visual Layerhere.