Package: opendatablend
Open Data Blend for PythonOpen Data Blend for Python is the fastest way to get data from the Open Data Blend Dataset API. It is a lightweight, easy-to-use extract and load (EL) tool.You can use theget_datafunction to download any data file belonging to an Open Data Blend dataset. Alternatively, theget_data_filesfunction can be used to download a collection of data files from an Open Data Blend dataset. The functions transparently download and cache the data locally or in cloud storage, mirroring the same folder hierarchy as on the remote server. They also cache a copy of the dataset metadata file (datapackage.json) at the point that they are called. The cache is persistent which means the files will be kept until they are deleted.The versioned dataset metadata can be used to re-download a specific version of a data file (sometimes referred to as 'time travel'). You can learn more about how we version our datasets in theOpen Data Blend Docs.In addition to downloading the data and metadata files,get_datareturns an object calledOutputwhich includes the locations of the downloaded files. Similarly,get_data_filesreturns an object calledOutputSetwhich includes the locations of the files that are downloaded and the associated metadata. From there, you can query and analyse the data directly using something light likePandasor, for more resource intensive processing, a data lakehouse platform likeDatabricks, or a scalable in-memory OLAP library likePolars.InstallationInstall the latest version ofopendatablendfromPyPI:pipinstallopendatablendUsage ExamplesNOTEIf you want to run the examples, be sure to replace placeholder values such as<ACCESS_KEY>with appropriate string literals or variables.Some of the following examples require thepandasandpyarrowpackages to be installed:pipinstallpandaspipinstallpyarrowMaking Public API RequestsNOTEPublic API requests have amonthly limit.Get the Dataimportopendatablendasodbimportpandasaspddataset_path='https://packages.opendatablend.io/v1/open-data-blend-road-safety/datapackage.json'# Specify the resource name of the data file. In this example, the 'date' data file will be requested in Parquet format.resource_name='date-parquet'# Get the data and store the output objectoutput=odb.get_data(dataset_path,resource_name)# Print the file locationsprint(output.data_file_name)print(output.metadata_file_name)Use The Data# Read a subset of the columns into a dataframedf_date=pd.read_parquet(output.data_file_name,columns=['drv_date_key','drv_date','drv_month_name','drv_month_number','drv_quarter_name','drv_quarter_number','drv_year'])# Check the contents of the dataframedf_dateMaking Authenticated API RequestsGet the Dataimportopendatablendasodbimportpandasaspddataset_path='https://packages.opendatablend.io/v1/open-data-blend-road-safety/datapackage.json'access_key='<ACCESS_KEY>'# Specify the resource name of the data file. In this example, the 'date' data file will be requested in Parquet format.resource_name='date-parquet'# Get the data and store the output objectoutput=odb.get_data(dataset_path,resource_name,access_key=access_key)# Print the file locationsprint(output.data_file_name)print(output.metadata_file_name)Use the Data# Read a subset of the columns into a dataframedf_date=pd.read_parquet(output.data_file_name,columns=['drv_date_key','drv_date','drv_month_name','drv_month_number','drv_quarter_name','drv_quarter_number','drv_year'])# Check the contents of the dataframedf_dateDownloading Multiple Data FilesTheget_data_filesfunction can be used to download a set of data files by providing their resource names as a list.Get the Dataimportopendatablendasodbimportpandasaspddataset_path='https://packages.opendatablend.io/v1/open-data-blend-road-safety/datapackage.json'access_key='<ACCESS_KEY>'# Specify the resource names of the data files. In this example, a subset of the available data files will be requested in Parquet format.resource_names=['date-parquet','time-of-day-parquet','geolocation-parquet','road-safety-accident-info-parquet','road-safety-accident-location-parquet','road-safety-accident-2021-parquet']# Get the data files and store the output objectoutput=odb.get_data_files(dataset_path,resource_names,access_key=access_key)# Print the file locationsprint(output.data_file_names)print(output.metadata_file_name)Ingesting Data Directly into Cloud Storage ServicesAzure Blob StorageUsingget_dataimportopendatablendasodbdataset_path='https://packages.opendatablend.io/v1/open-data-blend-road-safety/datapackage.json'access_key='<ACCESS_KEY>'# The access key can be set to an empty string if you are making a public API request# Specify the resource name of the data file. In this example, the 'date' data file will be requested in Parquet format.resource_name='date-parquet'# Get the data and store the output object using the Azure Blob Storage file systemconfiguration={"connection_string":"DefaultEndpointsProtocol=https;AccountName=<AZURE_BLOB_STORAGE_ACCOUNT_NAME>;AccountKey=<AZURE_BLOB_STORAGE_ACCOUNT_KEY>;EndpointSuffix=core.windows.net","container_name":"<AZURE_BLOB_STORAGE_CONTAINER_NAME>"# e.g. odbp-integration}output=odb.get_data(dataset_path,resource_name,access_key=access_key,file_system="azure_blob_storage",configuration=configuration)# Print the file locationsprint(output.data_file_name)print(output.metadata_file_name)Usingget_data_filesimportopendatablendasodbdataset_path='https://packages.opendatablend.io/v1/open-data-blend-road-safety/datapackage.json'access_key='<ACCESS_KEY>'# The access key can be set to an empty string if you are making a public API request# Specify the resource names of the data files. In this example, a subset of the available data files will be requested in Parquet format.resource_names=['date-parquet','time-of-day-parquet','geolocation-parquet','road-safety-accident-info-parquet','road-safety-accident-location-parquet','road-safety-accident-2021-parquet']# Get the data and store the output object using the Azure Blob Storage file systemconfiguration={"connection_string":"DefaultEndpointsProtocol=https;AccountName=<AZURE_BLOB_STORAGE_ACCOUNT_NAME>;AccountKey=<AZURE_BLOB_STORAGE_ACCOUNT_KEY>;EndpointSuffix=core.windows.net","container_name":"<AZURE_BLOB_STORAGE_CONTAINER_NAME>"# e.g. odbp-integration}output=odb.get_data_files(dataset_path,resource_names,access_key=access_key,file_system="azure_blob_storage",configuration=configuration)# Print the file locationsprint(output.data_file_names)print(output.metadata_file_name)Azure Data Lake Storage (ADLS) Gen2Usingget_dataimportopendatablendasodbdataset_path='https://packages.opendatablend.io/v1/open-data-blend-road-safety/datapackage.json'access_key='<ACCESS_KEY>'# The access key can be set to an empty string if you are making a public API request# Specify the resource name of the data file. In this example, the 'date' data file will be requested in Parquet format.resource_name='date-parquet'# Get the data and store the output object using the Azure Data Lake Storage Gen2 file systemconfiguration={"connection_string":"DefaultEndpointsProtocol=https;AccountName=<ADLS_GEN2_ACCOUNT_NAME>;AccountKey=<ADLS_GEN2_ACCOUNT_KEY>;EndpointSuffix=core.windows.net","container_name":"<ADLS_GEN2_CONTAINER_NAME>"# e.g. odbp-integration}output=odb.get_data(dataset_path,resource_name,access_key=access_key,file_system="azure_blob_storage",configuration=configuration)# Print the file locationsprint(output.data_file_name)print(output.metadata_file_name)Usingget_data_filesimportopendatablendasodbdataset_path='https://packages.opendatablend.io/v1/open-data-blend-road-safety/datapackage.json'access_key='<ACCESS_KEY>'# The access key can be set to an empty string if you are making a public API request# Specify the resource names of the data files. In this example, a subset of the available data files will be requested in Parquet format.resource_names=['date-parquet','time-of-day-parquet','geolocation-parquet','road-safety-accident-info-parquet','road-safety-accident-location-parquet','road-safety-accident-2021-parquet']# Get the data and store the output object using the Azure Data Lake Storage Gen2 file systemconfiguration={"connection_string":"DefaultEndpointsProtocol=https;AccountName=<ADLS_GEN2_ACCOUNT_NAME>;AccountKey=<ADLS_GEN2_ACCOUNT_KEY>;EndpointSuffix=core.windows.net","container_name":"<ADLS_GEN2_CONTAINER_NAME>"# e.g. odbp-integration}output=odb.get_data_files(dataset_path,resource_names,access_key=access_key,file_system="azure_blob_storage",configuration=configuration)# Print the file locationsprint(output.data_file_names)print(output.metadata_file_name)Amazon S3Usingget_dataimportopendatablendasodbdataset_path='https://packages.opendatablend.io/v1/open-data-blend-road-safety/datapackage.json'access_key='<ACCESS_KEY>'# The access key can be set to an empty string if you are making a public API request# Specify the resource name of the data file. In this example, the 'date' data file will be requested in Parquet format.resource_name='date-parquet'# Get the data and store the output object using the Amazon S3 file systemconfiguration={"aws_access_key_id":"<AWS_ACCESS_KEY_ID>","aws_secret_access_key":"AWS_SECRET_ACCESS_KEY","bucket_name":"<BUCKET_NAME>",# e.g. odbp-integration"bucket_region":"<BUCKET_REGION>"# e.g. eu-west-2}output=odb.get_data(dataset_path,resource_name,access_key=access_key,file_system="amazon_s3",configuration=configuration)# Print the file locationsprint(output.data_file_name)print(output.metadata_file_name)Usingget_data_filesimportopendatablendasodbdataset_path='https://packages.opendatablend.io/v1/open-data-blend-road-safety/datapackage.json'access_key='<ACCESS_KEY>'# The access key can be set to an empty string if you are making a public API request# Specify the resource names of the data files. In this example, a subset of the available data files will be requested in Parquet format.resource_names=['date-parquet','time-of-day-parquet','geolocation-parquet','road-safety-accident-info-parquet','road-safety-accident-location-parquet','road-safety-accident-2021-parquet']# Get the data and store the output object using the Amazon S3 file systemconfiguration={"aws_access_key_id":"<AWS_ACCESS_KEY_ID>","aws_secret_access_key":"AWS_SECRET_ACCESS_KEY","bucket_name":"<BUCKET_NAME>",# e.g. odbp-integration"bucket_region":"<BUCKET_REGION>"# e.g. eu-west-2}output=odb.get_data(dataset_path,resource_names,access_key=access_key,file_system="amazon_s3",configuration=configuration)# Print the file locationsprint(output.data_file_names)print(output.metadata_file_name)Google Cloud StorageUsingget_dataimportopendatablendasodbdataset_path='https://packages.opendatablend.io/v1/open-data-blend-road-safety/datapackage.json'access_key='<ACCESS_KEY>'# The access key can be set to an empty string if you are making a public API request# Specify the resource name of the data file. In this example, the 'date' data file will be requested in Parquet format.resource_name='date-parquet'# Get the data and store the output object using the Google Cloud Storage file systemconfiguration={"service_account_private_key_file":"<PATH_TO_SERVICE_ACCOUNT_PRIVATE_KEY_FILE>","bucket_name":"<BUCKET_NAME>",# e.g. odbp-integration"bucket_location":"<BUCKET_LOCATION>"# e.g. europe-west2}output=odb.get_data(dataset_path,resource_name,access_key=access_key,file_system="google_cloud_storage",configuration=configuration)# Print the file locationsprint(output.data_file_name)print(output.metadata_file_name)Usingget_data_filesimportopendatablendasodbdataset_path='https://packages.opendatablend.io/v1/open-data-blend-road-safety/datapackage.json'access_key='<ACCESS_KEY>'# The access key can be set to an empty string if you are making a public API request# Specify the resource names of the data files. In this example, a subset of the available data files will be requested in Parquet format.resource_names=['date-parquet','time-of-day-parquet','geolocation-parquet','road-safety-accident-info-parquet','road-safety-accident-location-parquet','road-safety-accident-2021-parquet']# Get the data and store the output object using the Google Cloud Storage file systemconfiguration={"service_account_private_key_file":"<PATH_TO_SERVICE_ACCOUNT_PRIVATE_KEY_FILE>","bucket_name":"<BUCKET_NAME>",# e.g. odbp-integration"bucket_location":"<BUCKET_LOCATION>"# e.g. europe-west2}output=odb.get_data(dataset_path,resource_names,access_key=access_key,file_system="google_cloud_storage",configuration=configuration)# Print the file locationsprint(output.data_file_names)print(output.metadata_file_name)OneLake in Microsoft FabricYou can use Open Data Blend for Python to ingest data directly into OneLake in Microsoft Fabric using a Fabric Notebook.PrerequisitesBefore attempting to ingest the data into OneLake using this method, you need to:Create aMicrosoft Fabric LakehouseCreate aMicrosoft Fabric NotebookSet adefault lakehousefor your Microsoft Fabric NotebookInstall the 'opendatablend' library through thePublic Librariessection of a new or exitingMicrosoft Fabric EnvironmentAttachthe environment to your workspace or notebookYou can then use the following methods to ingest the data. Pay special attention tobase_pathvalue because this is what controls where the data will be stored within OneLake. Thebase_pathmustpoint to the 'Files' location or a subfolder within it.Usingget_dataimportopendatablendasodbdataset_path='https://packages.opendatablend.io/v1/open-data-blend-road-safety/datapackage.json'access_key='<ACCESS_KEY>'# The access key can be set to an empty string if you are making a public API request# Specify the resource name of the data file. In this example, the 'date' data file will be requested in Parquet format.resource_name='date-parquet'# Specify the base path for where the data files should be landed. In this example, we want them to be stored in the root of the 'Files' folder in the Fabric lakehouse that has been attached to the Fabric notebook.base_path='/lakehouse/default/Files/'output=odb.get_data(dataset_path,resource_name,base_path=base_path,access_key=access_key)# Print the file locationsprint(output.data_file_name)print(output.metadata_file_name)Usingget_data_filesimportopendatablendasodbdataset_path='https://packages.opendatablend.io/v1/open-data-blend-road-safety/datapackage.json'access_key='<ACCESS_KEY>'# The access key can be set to an empty string if you are making a public API request# Specify the resource names of the data files. In this example, a subset of the available data files will be requested in Parquet format.resource_names=['date-parquet','time-of-day-parquet','geolocation-parquet','road-safety-accident-info-parquet','road-safety-accident-location-parquet','road-safety-accident-2021-parquet']# Specify the base path for where the data files should be landed. In this example, we want them to be stored in the root of the 'Files' folder in the Fabric lakehouse that has been attached to the Fabric notebook.base_path='/lakehouse/default/Files/'output=odb.get_data_files(dataset_path,resource_names,base_path=base_path,access_key=access_key)# Print the file locationsprint(output.data_file_names)print(output.metadata_file_name)Additional ExamplesFor more in-depth examples, see theexamplesfolder.