Package: azurebatchload
Azure Batch LoadHigh level Python wrapper for theAzure CLIto download or upload files in batches from or to Azure Blob Storage Containers.
This project aims to be themissing functionalityin the Python SDK of Azure Storage since there is no possibility to download or upload batches of files from or to containers.
The only option in the Azure Storage Python SDK is downloading file by file, which takes a lot of time.Besides doing loads in batches, since version0.0.5it's possible to set method tosinglewhich will use theAzure Python SDKto process files one by one.Installationpip install azurebatchloadSeePyPifor package index.Note: For batch uploads (method="batch") Azure CLI has to beinstalledandconfigured.
Check if Azure CLI is installed through terminal:az --versionRequirementsAzure Storage connection string has to be set as environment variableAZURE_STORAGE_CONNECTION_STRINGor
the seperate environment variablesAZURE_STORAGE_KEYandAZURE_STORAGE_NAMEwhich will be used to create the connection string.UsageDownload1. Using the standard environment variablesAzure-batch-load automatically checks for environment variables:AZURE_STORAGE_CONNECTION_STRING,AZURE_STORAGE_KEYandAZURE_STORAGE_ACCOUNT.
So if the connection_string or storage_key + storage_account are set as environment variables,
we can leave the argumentconnection_string,account_keyandaccount_nameempty:fromazurebatchloadimportDownloadDownload(destination='../pdfs',source='blobcontainername',extension='.pdf').download()2. Usingmethod="single"We can make skip the usage of theAzure CLIand just make use Python SDK by setting themethod="single":fromazurebatchloadimportDownloadDownload(destination='../pdfs',source='blobcontainername',extension='.pdf',method='single').download()3. Download a specific folder from a containerWe can download a folder by setting thefolderargument. This works both forsingleandbatch.fromazurebatchloadimportDownloadDownload(destination='../pdfs',source='blobcontainername',folder='uploads/invoices/',extension='.pdf',method='single').download()4. Download a given list of filesWe can give a list of files to download with thelist_filesargument.
Note, this only works withmethod='single'.fromazurebatchloadimportDownloadDownload(destination='../pdfs',source='blobcontainername',folder='uploads/invoices/',list_files=["invoice1.pdf","invoice2.pdf"],method='single').download()Upload:1. Using the standard environment variablesfromazurebatchloadimportUploadUpload(destination='blobcontainername',source='../pdf',extension='*.pdf').upload()2. Using themethod="single"method which does not require Azure CLI.fromazurebatchloadimportUploadUpload(destination='blobcontainername',source='../pdf',extension='*.pdf',method="single").upload()3. Upload a given list of files with thelist_filesargument.fromazurebatchloadimportUploadUpload(destination='blobcontainername',source='../pdf',list_files=["invoice1.pdf","invoice2.pdf"],method="single").upload()List blobsWith theUtils.list_blobsmethod we can do advanced listing of blobs in a container or specific folder in a container.
We have several argument we can use to define our scope of information:name_starts_with: This can be used to filter files with certain prefix, or to select certain folders:name_starts_with=folder1/subfolder/lastfolder/dataframe: Define if you want a pandas dataframe object returned for your information.extended_info: Get just the blob names or more extended information like size, creation date, modified date.1. List a whole container with just the filenames as a list.fromazurebatchloadimportUtilslist_blobs=Utils(container='containername').list_blobs()2. List a whole container with just the filenames as a dataframe.fromazurebatchloadimportUtilsdf_blobs=Utils(container='containername',dataframe=True).list_blobs()3. List a folder in a container.fromazurebatchloadimportUtilslist_blobs=Utils(container='containername',name_starts_with="foldername/").list_blobs()4. Get extended information a folder.fromazurebatchloadimportUtilsdict_blobs=Utils(container='containername',name_starts_with="foldername/",extended_info=True).list_blobs()5. Get extended information a folder returned as a pandas dataframe.fromazurebatchloadimportUtilsdf_blobs=Utils(container='containername',name_starts_with="foldername/",extended_info=True,dataframe=True).list_blobs()