Package: azure-datalake-store
Microsoft Azure Data Lake Store Filesystem Library for PythonThis project is the Python filesystem library for Azure Data Lake Store.INSTALLATIONTo install from source instead of pip (for local testing and development):>pipinstall-rdev_requirements.txt>pythonsetup.pydevelopUsage: Sample CodeTo play with the code, here is a starting point:fromazure.datalake.storeimportcore,lib,multithreadtoken=lib.auth(tenant_id,username,password)adl=core.AzureDLFileSystem(token,store_name=store_name)# typical operationsadl.ls('')adl.ls('tmp/',detail=True)adl.ls('tmp/',detail=True,invalidate_cache=True)adl.cat('littlefile')adl.head('gdelt20150827.csv')# file-like objectwithadl.open('gdelt20150827.csv',blocksize=2**20)asf:print(f.readline())print(f.readline())print(f.readline())# could have passed f to any function requiring a file object:# pandas.read_csv(f)withadl.open('anewfile','wb')asf:# data is written on flush/close, or when buffer is bigger than# blocksizef.write(b'important data')adl.du('anewfile')# recursively download the whole directory tree with 5 threads and# 16MB chunksmultithread.ADLDownloader(adl,"",'my_temp_dir',5,2**24)Progress can be tracked using a callback function in the formtrack(current, total)When passed, this will keep track of transferred bytes and be called on each complete chunk.Here’s an example using the Azure CLI progress controller as theprogress_callback:fromcli.core.applicationimportAPPLICATIONdef_update_progress(current,total):hook=APPLICATION.get_progress_controller(det=True)hook.add(message='Alive',value=current,total_val=total)iftotal==current:hook.end()...ADLUploader(client,destination_path,source_path,thread_count,overwrite=overwrite,chunksize=chunk_size,buffersize=buffer_size,blocksize=block_size,progress_callback=_update_progress)This will output a progress bar to the stdout:Alive[#########################                                       ]  40.0881%Finished[#############################################################]  100.0000%Usage: Command Line SampleTo interact with the API at a higher-level, you can use the provided
command-line interface in “samples/cli.py”. You will need to set
the appropriate environment variablesazure_usernameazure_passwordazure_data_lake_store_nameazure_subscription_idazure_resource_group_nameazure_service_principalazure_service_principal_secretto connect to the Azure Data Lake Store. Optionally, you may need to defineazure_tenant_idorazure_data_lake_store_url_suffix.Below is a simple sample, with more details beyond.pythonsamples\cli.pyls-lExecute the program without arguments to access documentation.To start the CLI in interactive mode, run “python samples/cli.py”
and then type “help” to see all available commands (similiar to Unix utilities):>pythonsamples/cli.pyazure>helpDocumentedcommands(typehelp<topic>):========================================catchmodclosedugethelplsmvquitrmdirtouchchgrpchowndfexistsheadinfomkdirputrmtailazure>While still in interactive mode, you can run “ls -l” to list the entries in the
home directory (“help ls” will show the command’s usage details). If you’re not
familiar with the Unix/Linux “ls” command, the columns represent 1) permissions,
2) file owner, 3) file group, 4) file size, 5-7) file’s modification time, and
8) file name.>pythonsamples/cli.pyazure>ls-ldrwxrwx---0123abcd0123abcd0Aug0212:44azure1-rwxrwx---0123abcd0123abcd1048576Jul2518:33abc.csv-r-xr-xr-x0123abcd0123abcd36Jul2218:32xyz.csvdrwxrwx---0123abcd0123abcd0Aug0313:46tmpazure>ls-l--human-readabledrwxrwx---0123abcd0123abcd0BAug0212:44azure1-rwxrwx---0123abcd0123abcd1MJul2518:33abc.csv-r-xr-xr-x0123abcd0123abcd36BJul2218:32xyz.csvdrwxrwx---0123abcd0123abcd0BAug0313:46tmpazure>To download a remote file, run “get remote-file [local-file]”. The second
argument, “local-file”, is optional. If not provided, the local file will be
named after the remote file minus the directory path.>pythonsamples/cli.pyazure>ls-ldrwxrwx---0123abcd0123abcd0Aug0212:44azure1-rwxrwx---0123abcd0123abcd1048576Jul2518:33abc.csv-r-xr-xr-x0123abcd0123abcd36Jul2218:32xyz.csvdrwxrwx---0123abcd0123abcd0Aug0313:46tmpazure>getxyz.csv2016-08-0418:57:48,603-ADLFS-DEBUG-Creatingemptyfilexyz.csv2016-08-0418:57:48,604-ADLFS-DEBUG-Fetch:xyz.csv,0-362016-08-0418:57:49,726-ADLFS-DEBUG-Downloadedtoxyz.csv,byteoffset02016-08-0418:57:49,734-ADLFS-DEBUG-Filedownloaded(xyz.csv->xyz.csv)azure>It is also possible to run in command-line mode, allowing any available command
to be executed separately without remaining in the interpreter.For example, listing the entries in the home directory:>pythonsamples/cli.pyls-ldrwxrwx---0123abcd0123abcd0Aug0212:44azure1-rwxrwx---0123abcd0123abcd1048576Jul2518:33abc.csv-r-xr-xr-x0123abcd0123abcd36Jul2218:32xyz.csvdrwxrwx---0123abcd0123abcd0Aug0313:46tmp>Also, downloading a remote file:>pythonsamples/cli.pygetxyz.csv2016-08-0418:57:48,603-ADLFS-DEBUG-Creatingemptyfilexyz.csv2016-08-0418:57:48,604-ADLFS-DEBUG-Fetch:xyz.csv,0-362016-08-0418:57:49,726-ADLFS-DEBUG-Downloadedtoxyz.csv,byteoffset02016-08-0418:57:49,734-ADLFS-DEBUG-Filedownloaded(xyz.csv->xyz.csv)>TestsFor detailed documentation about our test framework, please visit thetests folder.Need Help?Be sure to check out the Microsoft AzureDeveloper Forums on Stack Overflowif you have trouble with the provided code. Most questions are taggedazure and python.Contribute Code or Provide FeedbackIf you would like to become an active contributor to this project please
follow the instructions provided inMicrosoft Azure Projects Contribution Guidelines.
Furthermore, check outGUIDANCE.mdfor specific information related to this project.If you encounter any bugs with the library please file an issue in theIssuessection of the project.Code of ConductThis project has adopted theMicrosoft Open Source Code of Conduct.
For more information see theCode of Conduct FAQor contactopencode@microsoft.comwith any additional questions or comments.Release History0.0.53 (2023-04-11)Add MSAL support. Remove ADAL supportSuppress deprecation warning when detecting pyopenssl existence.0.0.52 (2020-11-25)Changed logging verbosity when closing a streamFilter out default acl for files when using recursive acl operations0.0.51 (2020-10-15)Add more logging for zero byte reads to investigate root cause.0.0.50 (2020-09-10)Fix bug with retrying for ADAL exception parsing.0.0.49 (2020-08-05)Fix bug with NoRetryPolicyRemove Python 3.4,5 in travis configuration.Fix logging for unicode0.0.48 (2019-10-18)Buffer writes to prevent out of memory problemsAdd Python 3.7 in travis configuration0.0.47 (2019-08-14)Remove logging of bearer tokenDocumentation related changes(Add readme.md and correct some formatting)0.0.46 (2019-06-25)Expose per request timeout. Default to 60.Concat will not retry by default.Bug fixes.0.0.45 (2019-05-10)Update open and close ADLFile semanticsRefactor code and improve performance for opening a file0.0.44 (2019-03-05)Add continuation token to LISTSTATUS api callUpdate api-version to 2018-09-010.0.43 (2019-03-01)Fix bug in downloader when glob returns a single file0.0.42 (2019-02-26)Update docstringsRemove logging setlevel to DEBUG for recursive acl operations0.0.41 (2019-01-31)Remove GetContentSummary api callMove check_token() under retry blockExpose timeout parameter for AdlDownloader and AdlUploaderRaise an exception instead of silently break for zero length reads0.0.40 (2019-01-08)Fix zero length readRemove dependence on custom wheel and conform to PEP 4200.0.39 (2018-11-14)Fix for Chunked Decoding exception thrown while reading response.content0.0.38 (2018-11-12)Added support for recursive acl functionsFixed bug due to missing filesessionid in get_chunk0.0.37 (2018-11-02)Reverted some changes introduced in 0.0.35 that didn’t work with other tokens0.0.36 (2018-10-31)Fixed typo in refresh_token call0.0.35 (2018-10-29)Added retry for getting tokensAdded requests>=2.20 because of CVE 2018-18074Fixed test parameters and updated test recordings0.0.34 (2018-10-15)Fixed concat issue with plus(or other symbols) in filenameAdded readinto methodChanged api-version to 2018-05-01 for all.0.0.32 (2018-10-04)Fixed test bugFixed empty folder upload bugFixed ADL Downloader block size bug0.0.31 (2018-09-10)Added support for batched ls0.0.30 (2018-08-28)Fixed .travis.yml order to add azure-nspg dependency0.0.29 (2018-08-22)Fixed HISTORY.rst and Pypi documentation0.0.28 (2018-08-20)Added recovery from DatalakeBadOffsetException0.0.27 (2018-08-08)Fixed bug in single file checkAdded Python2 exception compatibility0.0.26 (2018-08-03)Fixed bug due to not importing errnoFixed bug in os.makedirs race conditionUpdated Readme with correct environment variables and fixed some links0.0.25 (2018-07-26)Fixed downloading of empty directories and download of directory structure with only a single file0.0.24 (2018-07-16)Retry policy implemented for all operations, default being Exponential Retry Policy0.0.23 (2018-07-11)Fixed the incorrect download location in case of UNC local paths0.0.22 (2018-06-02)Encoding filepaths in URI0.0.21 (2018-06-01)Remove unused msrest dependency0.0.20 (2018-05-25)Compatibility of the sdist with wheel 0.31.00.0.19 (2018-03-14)Fixed upload issue where destination filename was wrong while upload of directory with single file #2080.0.18 (2018-02-05)Fixed read issue where whole file was cached while doing positional reads #198Fixed readline as well for the same0.0.17 (2017-09-21)Fixed README.rst indentation errorChanged management endpoint0.0.16 (2017-09-11)Fixed Multi chunk transfer hangs as merging chunks fails #187Added syncflag and leaseid in create, append calls.Added filesessionid in create, append and open calls.0.0.15 (2017-07-26)Enable Data Lake Store progress controller callback #174Fix File state incorrectly marked as “errored” if contains chunks is “pending” state #182Fix Race condition due totransferfuturedone_callback#1770.0.14 (2017-07-10)Fix an issue where common prefixes in paths for upload and download were collapsed into only unique paths.0.0.13 (2017-06-28)Add support for automatic refreshing of service principal credentials0.0.12 (2017-06-20)Fix a regression with ls returning the top level folder if it has no contents. It now properly returns an empty array if a folder has no children.0.0.11 (2017-06-02)Update to name incomplete file downloads with a.inprogresssuffix. This suffix is removed when the download completes successfully.0.0.10 (2017-05-24)Allow users to explicitly use or invalidate the internal, local cache of the filesystem that is built up from previouslscalls. It is now set to always call the service instead of the cache by default.Update to properly create the wheel package during build to ensure all pip packages are available.Update folder upload/download to properly throw early in the event that the destination files exist and overwrite was not specified. NOTE: target folder existence (or sub folder existence) does not automatically cause failure. Only leaf node existence will result in failure.Fix a bug that caused file not found errors when attempting to get information about the root folder.0.0.9 (2017-05-09)Enforce basic SSL utilization to ensure performance due toGitHub issue 625 <https://github.com/pyca/pyopenssl/issues/625>0.0.8 (2017-04-26)Fix server-side throttling retry support. This is not a guarantee that if the server is throttling the upload (or download) it will eventually succeed, but there is now a back-off retry in place to make it more likely.0.0.7 (2017-04-19)Update the build process to more efficiently handle multi-part namespaces for pip.0.0.6 (2017-03-15)Fix an issue with path caching that should drastically improve performance for download0.0.5 (2017-03-01)Fix for downloader to ensure there is access to the source path before creating destination filesFix for credential objects to inherit from msrest.authentication for more universal authentication supportAdd support for the following:set_expiry: allows for setting expiration on filesACL management:set_acl: allows for the full replacement of an ACL on a file or folderset_acl_entries: allows for “patching” an existing ACL on a file or folderget_acl_status: retrieves the ACL information for a file or folderremove_acl_entries: removes the specified entries from an ACL on a file or folderremove_acl: removes all non-default ACL entries from a file or folderremove_default_acl: removes all default ACL entries from a folderRemove unsupported and unused “TRUNCATE” operation.Added API-Version support with a default of the latest api version (2016-11-01)0.0.4 (2017-02-07)Fix for folder upload to properly delete folders with contents when overwrite specified.Fix to set verbose output to False/Off by default. This removes progress tracking output by default but drastically improves performance.0.0.3 (2017-02-02)Fix to setup.py to include the HISTORY.rst file. No other changes0.0.2 (2017-01-30)Addresses an issue with lib.auth() not properly defaulting to 2FAFixes an issue with Overwrite for ADLUploader sometimes not being honored.Fixes an issue with empty files not properly being uploaded and resulting in a hang in progress tracking.Addition of a samples directory showcasing examples of how to use the client and upload and download logic.General cleanup of documentation and comments.This is still based on API version 2016-11-010.0.1 (2016-11-21)Initial preview release. Based on API version 2016-11-01.Includes initial ADLS filesystem functionality and extended upload and download support.