Package: gbif-dl
gbif-dl 🌱 > 💾this package makes it simpler to obtain media data from the GBIF database to be used for trainingmachine learning classificationtasks. It wraps theGBIF APIand supports directly querying the api to obtain and download a list of urls.
Existing saved queries can also be obtained using the download api of GBIF simply by providing GBIF DOI key.
The package provides an efficient downloader that uses python asyncio modules to speed up downloading of many small files as typically occur in downloads.DisclaimerUnlike GBIF occurrences that all have a creative common license (CC0, CC BY, or CC BY-NC), GBIFdoes not give any official recommendationfor licensing shared media files. The License fields are essentially free text filled in by the data provider. Data providers are strongly encouraged to set their licenses in a machine-readable format, but there is no guarantee. Thus, it is the responsibility of GBIF-DL users to set up the appropriate filters on the license field and to respect the conditions of use of these licenses.Since we are heading into relatively new territory regarding the use of GBIF media files for machine learning, it is currently unclear how publishers will feel about this when learning that their photos are used in this way. At Pl@ntNet, the choice to publish our images in GBIF has been carefully considered and we are fully aware that they would probably be used for this purpose. But we fully understand that some other data providers didn't think of this and we are interested to have open discussion around these aspects.InstallationInstallation can be done via pip.pip install gbif-dlUsageThe usage ofgbif-dlhelps users to create their own GBIF based media pipeline for training machine learning models. The package provides two core functionalities as followed:gbif-dl.generators: Generators provide image urls from the GBIF database given queries or a pre-defined URL.gbif-dl.io: Provides efficient media downloading to write the data to a storage device.1. Retrieve media urls from GBIFgbif-dlsupports two ways to retrieve image urls. One is to use directly query the gbif api thegbif_dl.apimodule. This is suited for quickly retrieving smaller datasets that do not require extensive query parameters. Another way is to use already thegbif download workflowswhich assemble aDarwin Core Archiveswaiting on the gbif servers. These can be downloaded and parsed using thegbif_dl.dwcamodule as explained below.gbif_dl.generators.api: getting occurance media URLS by querying GBIFThe query supports all fields that are supported by theGBIF occurance API. In the following example, we query three plants using thespeciesKeyof GBIF from the list oftop 1200 invasive plant species. Also, we are limiting the results by only retrieving results fromPlantnetandiNaturalist. using thedatasetKey.The query is passed as a simple dictionary:queries={"speciesKey":[5352251,# "Robinia pseudoacacia L"3190653,# "Ailanthus altissima (Mill.) Swingle"3189866# "Acer negundo L"],"datasetKey":["7a3679ef-5582-4aaa-81f0-8c2545cafc81",# plantnet"50c9509d-22c7-4a22-a47d-8c48425ef4a7"# inaturalist]}Give this query, we can pass this to theapi.generate_urlsfunction which returns a python
generator:importgbif_dldata_generator=gbif_dl.api.generate_urls(queries=queries,label="speciesKey",)Additionally we have to specify the outputlabelfrom the occurances which doesn't
necessarily have to be part of the query attributes. Thelabelis later used to classify the results and store the data in hierachical structure:label/image.jpg.Iterating over the generator now yields the media data returning a few thousand urls.foriindata_generator:print(i)each return entry is a dictionary of media attributes, to be consumed by the downloader.{'url':'https://bs.plantnet.org/image/o/cfa25c7fb5cdf12719d1345769d3936d0ca73974','basename':'fdcc3440ab0e3abf824a5c68c864b018cccfcd3b','label':'5352251'},{'url':'https://static.inaturalist.org/photos/58881180/original.jpeg?1577914533','basename':'7db818c0708ba859516353ff9b30ef942aca19de','label':'3189866'},{'url':'https://static.inaturalist.org/photos/58866788/original.jpeg?1577898729','basename':'58ae3ef46e59e9a06d67de09c8b7ef3b8db3c85a','label':'3189866'}Balancing itemsVery often users won't be using all media downloads from a given query since this often results in datasets with heavily inbalanced number of samples per label. When generating urls from the API, users can specify certain additional attributes to influence the sampling process. For example, to balance the dataset by the dataset provider and by the species the following arguments can be used:split_streams_by: splits the query into combination of several substreams where each stream represents the product of the query values. When combined withnb_samples, this produces a balanced dataset where each stream yields the same number of samples.nb_samples: an integer that limits the total number of samples to be generated from the balanced streams. E.g, this can be used to just get100samples from the api. When set to-1, the minimum number of samples from all streams is used, hence this results in themaximum number of balancedsampled from all streams.In the following example, we will receive a balanced dataset assembled from3 species * 2 datasets = 6 streamsand only get minumum number of total samples from all 6 streams:data_generator=gbif_dl.api.generate_urls(queries=queries,label="speciesKey",nb_samples=-1,split_streams_by=["datasetKey","speciesKey"],)For other, more advanced, use-cases users can add more constraints:nb_samples_per_stream: put a hardlimiton themaximum number of samplesto be yielded by a stream.weighted_streams: weights each stream by its original distribution. That way users can get a smaller subset of the data but keep the originalunbalanceddistribution of the data.The following dataset consist of exactly 1000 samples for which the distribution ofspeciesKeyis maintained from the full query of all samples. Furthermore, we only allow a maxmimum of 800 samples per species.data_generator=gbifmediads.api.generate_urls(queries=queries,label="speciesKey",nb_samples=1000,nb_samples_per_stream=800,weighted_streams=True,split_streams_by=["speciesKey"],)Get URLS using Darwin Core ArchivesA url generator can also be created from a GBIF download link given a registered DOI or a GBIF download ID. In the following example we will be downloading and parse DWCA archivethat should yield the same results as in the query example above..dwca_root_path: Set root path where to store the DWCA zip files. Defaults to None, which results in the creation of a temporary directory, If the path and DWCA archive already exist, it will not be downloaded again.The following example creates a data_generator with the the same output class label as in the example above.data_generator=gbif_dl.dwca.generate_urls("10.15468/dl.vnm42s",dwca_root_path="dwcas",label="speciesKey")Downloading images to diskDownloading from a url generator can simply be done by running.stats=gbif_dl.io.download(data_generator,root="my_dataset")The downloader provides very fast download speeds by using an async queue. Some fail-safe functionality can be provided by setting the number ofretriesto higher than 1.Training DatasetsPyTorchgbif-dlmakes it simple to train a PyTorch image classification model by using e.g.torchvision.ImageFolder. Each item in thedata_generatorcan be randomly assigned to atrainortestsubset usingrandom_subsets. That way users can directly use the subsets.importtorchvisiongbif_dl.io.download(data_generator,root="my_dataset",random_subsets={'train':0.9,'test':0.1})train_dataset=torchvision.datasets.ImageFolder(root='my_dataset/train',...)test_dataset=torchvision.datasets.ImageFolder(root='my_dataset/test',...)TensorflowThe simpliest way to generate atf.data.Datasetpipeline from a data generator is to usetf.keras.preprocessing.image_dataset_from_directory.
Similarily to the pytorch example, users just need to provide the root paths of the downloaded datasets.importtensorflowastfgbif_dl.io.download(data_generator,root="my_dataset",random_subsets={'train':0.9,'test':0.1})train_dataset=tf.keras.preprocessing.image_dataset_from_directory(root='my_dataset/train',label_mode="categorical",labels="inferred",*args,**kwargs)test_dataset=tf.keras.preprocessing.image_dataset_from_directory(root='my_dataset/test',label_mode="categorical",labels="inferred",*args,**kwargs)FAQQ: Downloading doesn't work from inside a jupyter notebookThis is a known issue of running asyncio code from within jupyter.
Please execute these lines before using gbif-dlimportnest_asyncionest_asyncio.apply()LicenseMIT