Package: jupyter-duckdb
DuckDB Kernel for JupyterThis is a simple DuckDB wrapper kernel which accepts SQL as input, executes it
using a previously loaded DuckDB instance and formats the output as a table.
There are some magic commands that make teaching easier with this kernel.Quick StartTable of ContentsSetupUsing pipUsing DockerUsageA Note on Magic CommandsLoad a DatabaseSchema DiagramsNumber of RowsShip Tests With Your NotebookRelational AlgebraDomain CalculusSetupUsing pipRunpipto install the corresponding package frompypiafterJupyter is already installed.pipinstalljupyter-duckdbRegister the kernel.jupyterkernelspecinstall<pathtothesite-packagesdirectory>/duckdb_kernelNow start Jupyter the usual way and the kernel should be available.If DuckDB cannot be installed on your system, you can use SQLite as a backend
instead. To do this, set the environment variableSQLITEwhen running pip:SQLITE=1pipinstalljupyter-duckdbUsing DockerExecute the following command to pull and run a prepared image.dockerrun-p8888:8888troebs/jupyter-duckdbThis image can also be used with JupyterHub and theDockerSpawner / SwarmSpawnerand probably with thekubespawner. You can also build
your own image using theDockerfilein the repository.UsageA detailed example can be foundin the repository. The rest of this
section describes the magic commands.A Note on Magic CommandsMany Jupyter kernels make a difference between magic commands for a single line
starting with one percent sign and others for a whole cell starting with two
percent signs. The upcoming magic commands always apply to a whole cell.
Therefore, it does not matter whether you use a single or two percent signs.
However, the magic commands must always be used at the beginning of a cell.It is also possible to use more than one magic command per cell.Load a DatabaseTo load the database two magic commands are available.CREATEcreates a new database and therefore overwrites files with the same
name without prompting. Using the optional parameterOFyou can either provide
another DuckDB file or a file with SQL statements. In the first case the
included tables will be copied to the new database, while in the second case the
SQL statements are just executed. We find this feature very useful to work in a
temporary copy of the data and therefore be able to restart at any time. The
last optional parameterWITH_TESTSis described in
detailbelow.%CREATE data.duckdb OF my_statements.sqlLOADon the other hand loads an existing database and returns an error if it
does not exist. (That is whyOFcannot be used withLOAD!WITH_TESTSon
the other hand is available also with this magic command.)%LOAD data.duckdbOnly one database can be open at any time. If a new database is created or
loaded, the current one is closed first and saved to disk if necessary.Please note that:memory:is also a valid file path for DuckDB. The data is
then stored exclusively in the main memory. In combination withCREATEandOFthis makes it possible to work on a temporary copy in memory.Schema DiagramsThe magic commandSCHEMAcan be used to create a simple schema diagram of the
loaded database, showing all created tables, their columns and data types, but
without any views. Primary keys are printed in bold and unique keys are
underlined. Foreign keys are also highlighted and the dependencies between the
tables are shown by arrows.The optional flagTDcan be set to force a vertical layout. This
representation requires more space, but can improve readability.%SCHEMA TDGraphviz (dotin PATH) is required to render schema diagrams.Number of RowsBy default, only 20 rows are shown. All further lines are replaced by three
dots. When hovering over the three dots using the cursor, the number of omitted
lines is displayed. Of course, the number of lines displayed can be changed.The magic commandALL_ROWSand its short formALLcan be used to display *all* rows of the query in the same cell.Caution: With large result sets
this can lead to a frozen Jupyter instance.%ALL_ROWSSELECT*FROMfoo-- all rowsThe magic commandQUERY_MAX_ROWSfollowed by an integer can be used to change
the number of displayed rows for the current cell.%QUERY_MAX_ROWS50SELECT*FROMfoo-- 50 rowsThe magic commandMAX_ROWSfollowed by an integer can be used to change the
number of displayed rows for all future queries including the current cell.%MAX_ROWS30SELECT*FROMfoo-- 30 rowsSELECT*FROMbar-- 30 rowsShip Tests With Your NotebooksSimple tests can be loaded together with the database with the help of
theWITH_TESTSparameter. These tests are stored as a JSON file. Each test is
assigned a unique name, a result set and whether the test should check the order
of the result. A very simple test file looks like the following JSON object:{"task1":{"ordered":false,"equals":[[1,"Name 1"],[2,"Name 2"]]}}To bind a test to a cell, use the magic commandTESTin combination with a
name. After the cell is executed, the result is evaluated and then displayed
below the query result.%TESTtask1SELECT2,'Name 2'UNIONSELECT1,'Name 1'Disclaimer: The integrated testing is work-in-progress and thus subject to
potentially incompatible changes and enhancements.Relational AlgebraAn interpreter for relational algebra queries is integrated in this kernel. The
magic commandRAactivates the relational algebra mode for a single cell:%RA
π a, b (σ c = 1 (R))The supported operations are:ProjectionπSelectionσRenameβUnion∪Intersection∩Difference\Natural Join⋈Cross Product×Division÷The Dockerfile also installs the Jupyter Lab pluginjupyter-ra-extension. It adds
the symbols mentioned above and some other supported symbols to the toolbar for
insertion on click.Domain CalculusAn interpreter for domain calculus queries is integrated in this kernel. The
magic commandDCactivates the domain calculus mode for a single cell:%DC
{ a, b | R(a, b, c) ∧ c = 1 }