Package: dbfs-package-sync
dbfs-package-syncScenario: You are developing a Python package that you need in a couple of notebooks.
You (of course) want to develop your package in an IDE.
You can write unit tests and perhaps schedule some notebooks to run integration tests,
however, during the development process you also want to prototype and run your in-development-code on a cluster.There is the extended CLI tool for Databricks (dbx) that has syncing functionality, but the problem there is that
you have to have your code as a Repo inside Databricks. There is also a plugin for Visual Studio Code,
but maybe you don't want to use VSC and there too you need your repo linked to Databricks.So what is the problem with linking your repo to Databricks, it's convenient right? If you place a test-notebook
at the same level as where your package folder is located you can import it without the need for activatingautoreloador setting the system path.
That is a great feature that gives me hope for Databricks in the future, but
sometimes you have your Python package as part of a larger repo that you do not necessarily
want in Databricks.Also, Databricks still does not support Poetry, so if you want to install the dependencies of your package on the
cluster, you first need to install poetry, create a requirements file and install the dependencies using%pip.I am basically naming a few excuses to justify me making this package for myself. I just want to sync my package
to dbfs, no more, no less. Why all the fuss?So, that's what this CLI does. When I need to test the current state of my package in some Databricks notebook,
I rundbfspsand I have my local state on DBFS.Caveats:Only works for poetry packagesAt the top of the notebook you will of course need to install the dependencies once and setautoreload.
Thedbfspscommand will send along a requirements file and create a helper notebook though, so you only need a single%runcommand at the top of your notebook.I opted for running a single command every time you need to sync your code instead of a continuous syncing process
running in the background. To keep track of which files need to be uploaded/removed,dbfspscreates a hidden
text file.dbfsps_file_status.ExampleIf I want to sync the package of this repo, I would run the following from the rootpoetryrundbfsps\--profilesome-profile-dev\--package-locationdbfsps\--remote-pathdbfs:/FileStore/jmeidam/packages\dbfs-package-syncThis would create a remote folder atdbfs:/FileStore/jmeidam/packages/dbfs_package_sync. This folder would contain
arequirements.txtfile and a folder named "dbfsps" with the contents of that directory in this repo.I can then upload theinit_dbfs_package_sync.pynotebook and run it at the top of any other notebook that uses
imports from thedbfspspackage:%run init_dbfs_package_syncDatabricks CLIdbfspsmakes use of Databricks Command Line Interface. To be able to sync your package with DBFS, you will need to
set up a databricks-cli profile.If you haven't done so already, generate a token on Databricks (can be found under user settings).
Store the token somewhere and rundatabricksconfigure--profilesome-profile-dev--tokenThis will ask for that token and the host, e.g.https://adb-8765432123456789.12.azuredatabricks.net.You will need the profile name in the--profileoption ofdbfspsMore detailed documentation on the CLI can be found here:https://docs.databricks.com/user-guide/dev-tools/databricks-cli.html#databricks-cliNotesThe following three files are generated by thedbfspscommand, you may want to add those to your.gitignorefile:.dbfsps_file_statusrequirements.txtinit_<package_name>.py