Package: cdsetool
CDSEToolAbout CDSEToolThis script downloads copernicus data from the Copernicus Data Space EcosystemQuick startfromcdsetool.queryimportquery_features,shape_to_wktfromcdsetool.credentialsimportCredentialsfromcdsetool.downloadimportdownload_featuresfromcdsetool.monitorimportStatusMonitorfromdatetimeimportdatefeatures=query_features("Sentinel1",{"startDate":"2020-12-20","completionDate":date(2020,12,25),"processingLevel":"LEVEL1","sensorMode":"IW","productType":"IW_GRDH_1S","geometry":shape_to_wkt("path/to/shapefile.shp"),},)list(download_features(features,"path/to/output/folder/",{"concurrency":4,"monitor":StatusMonitor(),"credentials":Credentials("username","password"),},))Or use the CLI:cdsetoolquerysearchSentinel2--search-termstartDate=2020-01-01--search-termcompletionDate=2020-01-10--search-termprocessingLevel=S2MSI2A--search-termbox="4","51","4.5","52"cdsetooldownloadSentinel2PATH/TO/DIR--concurrency4--search-termstartDate=2020-01-01--search-termcompletionDate=2020-01-10--search-termprocessingLevel=S2MSI2A--search-termbox="4","51","4.5","52"Table of ContentsCDSEToolAbout CDSEToolQuick StartTable of ContentsInstallatsonUsageQuerying featuresQuerying by shapesQuerying by lists of parametersQuerying by datesListing search termsDownloading featuresAuthenticatingConcurrently downloading featuresSequentially downloading featuresRoadmapContributingLICENSEInstallationInstallcdsetoolusing pip:pipinstallcdsetool==0.2.4UsageQuerying featuresQuerying is always done in batches, returninglen(results) <= maxRecordsrecords each time.
A local buffer is filled and gradually emptied as results are yielded. When the buffer is empty,
more results will be requested and the process repeated until no more results are available, or
the iterator is discarded.Since downloading features is the most common use-case,query_featuresassumes that the query will run till the end.
Because of this, the batch size is set to2000, which is the size limit set by CDSE.fromcdsetool.queryimportquery_featurescollection="Sentinel2"search_terms={"maxRecords":"100",# batch size, between 1 and 2000 (default: 2000)."startDate":"1999-01-01","processingLevel":"S2MSI1C"}# wait for a single batch to finish, yield results immediatelyforfeatureinquery_features(collection,search_terms):# do something with feature# wait for all batch requests to complete, returning listfeatures=list(query_features(collection,search_terms))# manually iterateiterator=query_features(collection,search_terms)featureA=next(iterator)featureB=next(iterator)# ...Querying by shapesTo query by shapes, you must first convert your shape to Well Known Text (WKT). The includedshape_to_wktcan solve this.fromcdsetool.queryimportquery_features,shape_to_wktgeometry=shape_to_wkt("path/to/shape.shp")features=query_features("Sentinel3",{"geometry":geometry})Querying by lists of parametersMost search terms only accept a single argument. To query by a list of arguments, loop the arguments
and pass them one by one to the query function.fromcdsetool.queryimportquery_featurestile_ids=["32TPT","32UPU","32UPU","31RFL","37XDA"]fortile_idintile_ids:features=query_features("Sentinel2",{"tileId":tile_id})forfeatureinfeatures:# do things with featureQuerying by datesIts quite common to query for features created before, after or between dates.fromcdsetool.queryimportquery_featuresfromdatetimeimportdate,datetimedate_from=date(2020,1,1)# or datetime(2020, 1, 1, 23, 59, 59, 123456) or "2020-01-01" or "2020-01-01T23:59:59.123456Z"date_to=date(2020,12,31)features=query_features("Sentinel2",{"startDate":date_from,"completionDate":date_to})Listing search termsTo get a list of all search terms for a given collection, you may either use thedescribe_collectionfunction or
use the CLI:fromcdsetool.queryimportdescribe_collectionsearch_terms=describe_collection("Sentinel2").keys()print(search_terms)And the CLI:$cdsetoolquerysearch-termsSentinel2Downloading featuresAuthenticatingAn account is required to download features from the Copernicus distribution service.To authenticate using an account, instantiateCredentialsand pass your username and passwordfromcdsetool.credentialsimportCredentialsusername="konata@izumi.com"password="password123"credentials=Credentials(username,password)Alternatively,Credentialscan pull from~/.netrcwhen username and password are left blank.# ~/.netrcmachinehttps://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/tokenloginkonata@izumi.compasswordpassword123# main.pyfromcdsetool.credentialsimportCredentialscredentials=Credentials()The credentials object may then be passed to a download function. If left out, the download
functions will default to using.netrc.credentials=Credentials()download_features(features,"/some/download/path",{"credentials":credentials})Credentials can be validated using thevalidate_credentialsfunction which will return a boolean.fromcdsetool.credentialsimportvalidate_credentialsvalidate_credentials(username='user',password='password')If None are passed to username and password,validate_credentialswill validate.netrcConcurrently downloading featuresCDSETool provides a method for concurrently downloading features. The concurrency level
should match your accounts privileges.
SeeCDSE quotasThe downloaded feature ids are yielded, so its required to await the results.fromcdsetool.queryimportquery_featuresfromcdsetool.downloadimportdownload_featuresfeatures=query_features("Sentinel2")download_path="/path/to/download/folder"downloads=download_features(features,download_path,{"concurrency":4})foridindownloads:print(f"feature{id}downloaded")# orlist(downloads)Sequentially downloading featuresIts possible to download features sequentially in a single thread if desired.fromcdsetool.queryimportquery_featuresfromcdsetool.downloadimportdownload_featurefeatures=query_features("Sentinel2")download_path="/path/to/download/folder"forfeatureinfeatures:download_feature(feature,download_path)RoadmapQuery schema validationHigh-level APIQuery featuresDownload featuresDownload single featureDownload list of featuresDownload by IDDownload by URLCommand-Line InterfaceUpdate to match the high-level APIBetter--helpmessagesQuickstart guide in README.mdTest suiteQueryCredentialsDownloadMonitorStrategy for handling HTTP and connection errorsContributingAny contributions you make aregreatly appreciated.If you have a suggestion that would make this better, please fork the repo and create a pull request.
You can also simply open an issue with the tag "enhancement".Fork the ProjectCreate your Feature Branch (git checkout -b feature/cool-new-feature)Commit your Changes (git commit -m 'Add some feature')Push to the Branch (git push origin feature/cool-new-feature)Open a Pull RequestLICENSEDistributed under the MIT License. SeeLICENSEfor more information.