Package: blackbricks
BlackbricksA formatting tool for your Databricks notebooks.Python cells are formatted withblackSQL cells are formatted withsqlparseTable of ContentsInstallationUsageVersion control integrationContributingFAQBreaking changesInstallationWhile you can usepipdirectly, you should prefer usingpipx.$pipxinstallblackbricksYou probably also want to have installed thedatabricks-cli, in order to useblackbricksdirectly on your notebooks.$pipxinstalldatabricks-cli
$databricksconfigure# Required in order to use `blackbricks` on remote notebooks.UsageYou can useblackbrickson Python notebook files stored locally, or directly on the notebooks stored in Databricks.For the most part,blackbricksoperates very similarly toblack.$blackbricksnotebook1.pynotebook2.py# Formats both notebooks.$blackbricksnotebook_directory/# Formats every notebook under the directory (recursively).An important difference is thatblackbrickswill ignore any file that does not contain the# Databricks notebook sourceheader on the first line. Databricks adds this line to all Python notebooks. This means you can happily runblackbrickson a directory with both notebooks and regular Python files, andblackbrickswon't touch the latter.If you specify the-ror--remoteflag,blackbrickswill work directly on your notebooks stored in Databricks.$blackbricks--remote/Users/username/notebook.py
$blackbricks--remote/Repos/username/repo-name/notebook.pyFull usage$ poetry run blackbricks --help

 Usage: blackbricks [OPTIONS] [FILENAMES]...

 Formatting tool for Databricks python notebooks.
 Python cells are formatted using `black`, and SQL cells are formatted by `sqlparse`.
 Local files (without the `--remote` option):
 - Only files that look like Databricks (Python) notebooks will be processed. That is,
 they must start with the header `# Databricks notebook source`
 - If you specify a directory as one of the file names, all files in that directory will
 be added, including any subdirectory.
 Remote files (with the `--remote` option):
 - Make sure you have installed the Databricks CLI (``pip install databricks_cli``)
 - Make sure you have configured at least one profile (`databricks configure`). Check the
 file `~/.databrickscfg` if you are not sure.
 - File paths should start with `/`. Otherwise they are interpreted as relative to
 `/Users/username`, where `username` is the username specified in the Databricks profile
 used.

╭─ Arguments ────────────────────────────────────────────────────────────────────────────╮
│   filenames      [FILENAMES]...  Path to the notebook(s) to format. [default: None]    │
╰────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ──────────────────────────────────────────────────────────────────────────────╮
│ --remote                     -r                             If this option is used,    │
│                                                             all filenames are treated  │
│                                                             as paths to notebooks on   │
│                                                             your Databricks host (i.e. │
│                                                             not local files).          │
│ --profile                    -p                    NAME     If using --remote, which   │
│                                                             Databricks profile to use. │
│                                                             [default: DEFAULT]         │
│ --line-length                                      INTEGER  How many characters per    │
│                                                             line to allow.             │
│                                                             [default: 88]              │
│ --sql-upper                      --no-sql-upper             SQL keywords should be     │
│                                                             UPPERCASE or lowercase.    │
│                                                             [default: sql-upper]       │
│ --check                                                     Don't write the files      │
│                                                             back, just return the      │
│                                                             status. Return code 0      │
│                                                             means nothing would        │
│                                                             change.                    │
│ --diff                                                      Don't write the files      │
│                                                             back, just output a diff   │
│                                                             for each file on stdout.   │
│ --version                                                   Display version            │
│                                                             information and exit.      │
│ --help                                                      Show this message and      │
│                                                             exit.                      │
╰────────────────────────────────────────────────────────────────────────────────────────╯Version control integrationUsepre-commit. Add a.pre-commit-config.yamlfile
to your repo with the following content (changing/removing theargsas you
wish):repos:-repo:https://github.com/inspera/blackbricksrev:1.0.0hooks:-id:blackbricksargs:[--line-length=120]Set therevattribute to the most recent version ofblackbricks.
Theargsare optional and can be used to set any ofblackbricksoptions.ContributingIf you find blackbricks useful, feel free to say so with a star. If you think it is utterly broken, you are more than
welcome to contribute improvements. Please open an issue first to discuss what you want added/fixed. Unless you are just
adding tests. In that case your pull request is extremely likely to be merged right away.FAQCan I disable SQL formatting?Sure! Certain SQL statements might not be parsed and indented properly bysqlparse, and the result can be jumbled
formatting. You can disable SQL formatting for a cell by adding-- nofmtto the very first line of a cell:%sql-- nofmtselectthis,sql_will,-- be kept justlike_thisfromif_that_is.what_you_needHow do I useblackbrickson my Databricks notebooks?First, make sure you have set updatabricks-clion your system (seeinstallation), and that you have
at least one profile setup in~/.databrickscfg. As an example:# File: ~/.databrickscfg[DEFAULT]host=https://dbc-b23456-a1243.cloud.databricks.com/username=username@example.compassword=dapi12345678901234567890[OTHERPROFILE]host=https://dbc-c54321-d234.cloud.databricks.comusername=name.user@example.compassword=dapi09876543211234567890You should useaccess tokensinstead of your actual password.You can then do:$blackbricks--remote/Users/username@example.com/notebook.py# Uses DEFAULT profile.$blackbricks--remotenotebook.py# Equivalent to the above.$blackbricks--remote--profileOTHERPROFILE/Users/name.user@example.com/notebook.py
$blackbricks--remote--profileOTHERPROFILEnotebook.py# Equivalent to the above.$blackbricks--remote/Repos/username@example.com/repo-name/notebook.py# Targeting notebook in a RepoCan you run blackbricks while using Databricks in the browser?No. Seehttps://github.com/inspera/blackbricks/issues/27for why.However, Databricks now allows you toformat your notebooks with black directly.I get an error:TypeError: init() got an unexpected keyword argument 'no_args_is_help'This means you had an old version ofclickinstalled from before, and your installation didn't upgrade it
automatically. Updating your installation should do the trick, e.g.pip install -U blackbricksor similar depending on
your installation method of choice.Shell commands like!lsthrows an errorSeehttps://github.com/inspera/blackbricks/issues/21.Breaking changesVersion policyStyle choices made byblackbrickswill follow semantic versioning, with changes that cause differences resulting in
new major versions. Such changes will be kept to an absolute minimum, with none currently planned.Style choices made byblack(responsible for 95% of the formatting in a notebook) will not follow the same strict
semantic versioning. This is becauseblackitself does not use semver, but instead provide ayear-based
policy.blackbrickswill
make aminorversion increase when it upgrades black to a new year. Such a bump should be made once the new year's
release ofblackis available. Feel free to open an issue if this has not been done yet.Breaking changes with version 2.0Notebooks will be terminated with a\nstarting with version2.0.0. This harmonizes EOF handling and should be much
less annoying in practice than prior versions. This causes a diff onanynotebook that was previously formatted withblackbricks<2.0.0.Also, the deprecated and non-functional flag for two space indentation is removed, and providing said flag is now an error.Breaking changes with version 1.0Earlier versions of blackbricks applied a patched version of black in order to allow two-space indentation. This was
done because Databricks used two-space indentation, and did not allow you to change that.Since then, Databricks has added the option to choose. Because you can now choose, blackbricks re-joins black in being
uncompromising, and since version 1.0 you can no longer choose anything but 4 space indentation.If youmustkeep using two-space indentation, then stick to versions<1.0.