Package: fastdbfs
Introductionfastdbfsis an interactive command line client for Databricks DBFS
API with the aim of being fast, friendly and feature rich.fastdbfsis still in alpha state. Anything may change at this
point. You should expect bugs and even data corruption or data
lost at times.Comments, bug reports, ideas and patches or pull requests are very
welcome.InstallationAs development offastdbfsprogresses, from time to time and at
points where it is considered to be more or less stable, releases on
PyPi are done.Those versions can be installed usingpipas follows:pip install fastdbfsBut at this early development stage, gettingfastdbfsdirectly from
GitHub is still probably a better idea, even if it may be broken at
times, you will enjoy newer features.You can do it as follows:git clone https://github.com/salva/fastdbfs.git
cd fastdbfs
python setup.py install --userUsageOnce the program is installed, just invoke it from the command line asfastdbfs.I don't recomend at this point the usage of the python modules
directly as the interfaces are not stable yet.Configurationfastdbfsreads its configuration from~/.config/fastdbfsand~/.databrickscfg(on windows that translates to something likeC:\Users\migueldcsa\.config\fastdbfsandC:\Users\migueldcsa\.databrickscfgrespectively).The official Databricks documentation contains instructions on how tocreate a
tokenand how toobtain the cluster
ID.Sample configuration[fastdbfs]
pager=less
editor=vi

[logging]
filename=/tmp/fastdbfs.log
level=INFO

[DEFAULT]
host = https://westeurope.azuredatabricks.net
token = dapi1234567890abcdef1234567890abcdef
cluster_id = 0111-222222-yoooh666

[other-environment]
host = https://westeurope.azuredatabricks.net
token = dapi0123456789abcdef0123456789abcdef
cluster_id = 0222-222222-yoooh666A commented version of this configuration file is availablehere.CommandsOncefastdbfsis launched, a prompt appears and the following
commands can be used:open [profile]Sets the active Databricks profile used for communicating.By default it usesDEFAULT.cd [directory]Changes the remote current directory.lcd [directory]Sets the local working directory.lpwdShows the working local directoryls [OPTS] [directory]:List the contents of the remote directory.The supported options are as follows:-l,--long: Include file properties size and modification time.-h,--human: Print file sizes in a human friendly manner.find [OPTS] [RULES] [directory]List files recursively according to a set of rules.Supported options are as follows:--nowarn: Do not report errors happening while walking the file
system tree.-l,--ls: Display file and directory properties.-h,--human: Show sizes like1K,210M,4G, etc.In addition to those, find also supports the following set of
options for selectively picking the files that should be
displayed:--min-size=SIZE,--max-size=SIZE: Picks files according to their
size (this rule is ignored for directories).--newer-than=date,--older-than=date: Picks entries according to
their modification time.--name=GLOB_PATTERN: Picks only files and directories whose
basename matches the given glob pattern.--iname=GLOB_PATTERN: Case insensitive version ofname.--re=REGULAR_EXPRESSION: Picks only entries whose basename matches
the given regular expression.--ire=REGULAR_EXPRESSION: Case insensitive version ofre.--wholere=REGULAR_EXPRESSION: Picks file names whose relative path
matches the given regular expression.--iwholere=REGULAR_EXPRESSION: Case insensitive version ofwholere.--external-filter=CMD: Filters entries using an external command.Also, any of the rules above can be negated preceding it by--exclude, for instance--exclude-iname=*.jpegThe rules above never cut the file system traversal. So for instance,
a rule discarding some subdirectory, doesn't preclude that
subdirectory for being traversed and its child entries picked.The basename is the last component of a patch. Relative paths are
considered relative to the root directory passed as an argument.Example:find --ls --newer-than=yesterday --max-size=100K --iname=*.jpgput src [dest]Copies the given local file to the remote system.Supported options are:-o,--overwrite: When a file already exists at the target
location, it is overwritten.get [OPTS] src [dest]Copies the given remote file to the local system.Supported options are as follows:-o,--overwrite: When a file already exists at the target
location, it is overwritten.rget [OPTS] [RULES] [src [target]]Copies the given remote directory to the local system
recursively.The options supported are as follows:-v,--verbose: Display the names of the files being copied.--nowarn: Do not show warnings.-o,--overwrite: Overwrite existing files.--sync: Copy only files that have changed.Before transferring a file it is checked whether a local file
already exists at the destination, if it is as new as the remote one
and if the sizes are the same. The download is skipped whan all
these conditions are true.In addition to those,rgetalso accepts the same set of predicates
asfindfor selecting the entries to copy.Examples:rget --iname *.png --exclude-iwholere=/temp/ . /tmp/pngsrput [OPTS] [src [dest]]Copies the given local directory to the remote system
recursively.Supported options are:-o,--overwrite: Overwrites any previously existent remote file.rm [OPTS] pathDelete the remote file or directory.Supported options are as follows:-R,--recursive: Delete files and directories recursively.mkdir dirCreates the directory (and any required parents).mkcd dirCreates the directory and sets it as the working directory.cat filePrints the contents of the remote file.show [OPTS] fileShows the contents of the remote file using the configured pager (for
instance,more).The supported options are as follows:--pager=PAGER: Picks the pager for displaying the file contents.The pager can be configured adding an entrypagerinside thefastdbfssection in the configuration file.lessis used by
default.The commandsbatcat,lessandmoreare shortcuts that use the
corresponding pager.edit [OPTS] fileRetrieves the remote file and opens it using your favorite editor.Once the editor is closed, the file is copied back to the remote
systemThe supported options are as follows:-n,--new: Creates a new file.--editor=EDITOR: Picks the editor.By defaultfastdbfspicks the editor from the environment variableEDITOR. It can also be customized in the configuration file creating
aneditorentry inside thefastdbfssection.The commandsviandmgare
shortcuts foreditthat will use the corresponding editors.!cmd ...Runs the given command locally.exitExits the client.External filtersTODOLimitationsDevelopment is primarily done on Linux and only from time to time isfastdbfstested on Windows. Don't hesitate to report bugs related
to this.The DBFS API has some limitations thatfastdbfscan not overcome:Directory listings timeout after 1min.The API has a throttling mechanism that slows down operations that
require a high number of calls (i.e. find, rput, rget).The methods provided for uploading data are too simplistic. They can
not be parallelized and in some edge cases transfers may become
corrupted (fastdbfstries to protect against that).The metadata available is limited to file size and modification
time.Glob expression checking is done using pythonfnmatchmodule
that only supports a very small set of patterns. Specifically, it
lacks support for alternations as in*.{jpeg,jpg,png}.TODOAdd more commands: mget, mput, glob, etc.Improve command line parsing allowing for command flags, pipes,
redirections, etc.Autocomplete.Make history persistent between sessions.Allow passing commands when the program is launched (for instance,
setting the default profile).Catch C-c during long tasks and perform an orderly cleanup
(i.e. remove temporary files).Improve code quality.Development and supportThe source code for this program is available fromhttps://github.com/salva/fastdbfs.You can also use the GitHub bug-tracker to report any issues.See alsoDatabricksCLI
tooland thedbfsshortcut. The
PyPipage.Databricks
connect. The
PyPipage.The documentation for theDBFS
APIthatfastdbfscalls under the hood.DBFS Explorer for
Databricksis
another client for DBFS providing a graphical user
interface.Projectat GitHub.CopyrightCopyright (C) 2021 Salvador Fandiño García (sfandino@yahoo.com)This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or (at
your option) any later version.This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
General Public License for more details.You should have received a copy of the GNU General Public License
along with this program.  If not, seehttps://www.gnu.org/licenses/.