Package: datasette-parquet
datasette-parquetSupport DuckDB, Parquet, CSV andJSON Linesfiles in Datasette. Depends on DuckDB.There is a demo athttps://dux.fly.dev/parquetCompare a query usingParquet on DuckDBvs the same query onSQLite. The DuckDB query is ~3-5x faster. On a machine with more than 1 core, DuckDB would outperform by an even higher margin.InstallationInstall this plugin in the same environment as Datasette.datasette install datasette-parquetUsageYou can use this plugin to access a DuckDB file, or a directory of CSV/Parquet/JSON files.DuckDB fileTo mount the/data/mydb.duckdbfile as a database calledmydb, create a metadata.json like:{
  "plugins": {
    "datasette-parquet": {
      "mydb": {
        "file": "/data/mydb.duckdb"
      }
    }
  }
}Directory of CSV/Parquet/JSON filesSay you have a directory of your favourite CSVs, newline-delimited JSON and parquet
files that looks like this:/data/census.csv
/data/books.tsv
/data/tweets.jsonl
/data/geonames.parquet
/data/sales/january.parquet
/data/sales/february.parquetYou can expose these in a Datasette database calledtroveby something
like this in yourmetadata.json:{
  "plugins": {
    "datasette-parquet": {
      "trove": {
        "directory": "/data",
        "watch": true
      }
    }
  }
}Then launch Datasette viadatasette --metadata metadata.jsonYou will have 5 views in thetrovedatabase:census,books,tweets,geonamesandsales.
Thesalesview will be the union of all the files in that directory -- this works for all of the file types, not just Parquet.Because you passed thewatchoption with a value oftrue, Datasette will automatically discover when
files are added or removed, and create or remove views as needed.Common optionsThese options can be used in either mode.httpfs- set totrueto enable theHTTPFS extensionCaveatsWarningYou know that old canard, that if it walks like a duck and quacks like a duck, it's probably a duck? This plugin tries to teach DuckDB to walk like SQLite and talk like SQLite. It's difficult, and frankly, I just winged this part. If you come across broken features, let me know and I'll try to fix them up.No timeouts: A core feature of Datasette is that it's safe to let the unwashed masses run arbitrary queries. This is because the data is immutable, and there are timeouts to prevent runaway CPU usage. DuckDB does not currently support timeouts. Think carefully about letting anonymous users use a Datasette instance with this plugin.You will likely want todisable facet suggestions from the CLI, or installdatasette-ui-extras, which disables facet suggestions.Joining with existing data: This plugin uses DuckDB, not SQLite. This means that you cannot join against your existing SQLite tables.Read-only: the data in the files can only be queried, not changed.Performance: the files are queried in-place. Performance will be limited by the file type -- parquet files have a zippy binary format, but large CSV and JSONL files might be slow.Facets: DuckDB supports a different set of syntax than SQLite. This means some Datasette features are incompatible, and will be disabled for DuckDB-backed files.Technical notesThis plugin has a mix of accidental complexity and essential complexity.
The essential complexity comes from things like "DuckDB supports a different
dialect of SQL". The accidental complexity comes from things like "it's
called theLawof Demeter, Colin, not the Strongly Held Opinion
of Demeter".This is a loose journal of things I ran into:DuckDB's Python API is similar to thesqlite3module's interface, but not
the same. Datasette expects to talk to an interface that conforms tosqlite3,
so this plugin crufts up some proxy objects to give a "convincing" facade.
I mostly YOLOd this part. I wouldn't trust it for write queries, or for
reading sensitive data.DuckDB doesn't have the concept of a separate cursor class.sqlite3's cursor is an iterableDatasette uses sqlite3.Row objects, which support indexing by namesqlite3 supports parameterized queries likeexecute('SELECT :p', {'p': 123}).
These need to be rewritten to use numbered parameters and a list.SQLite supports slightly different syntax than DuckDB. We usesqlglotto transpile queries into DuckDB's dialect.In homage to MySQL, SQLite supports string literals delimited by double
quotes. Datasette uses this feature, seehttps://github.com/simonw/datasette/issues/2001In homage to SQL Server, SQLite supports quoting identifiers with square
brackets. Datasette uses this feature, seehttps://github.com/simonw/datasette/issues/2013Unfortunately, using sqlglot brings its own challenges: it doesn't recognize
theGLOBoperator, seehttps://github.com/tobymao/sqlglot/issues/1066Datasette passes extraneous parameters to the sqlite3 connection. A writable
canned query will post acsrftokenfor security purposes, which ends up
as part of the query parameters. DuckDB is strict on the parameters matching
the SQL query, so it fails.Datasette expects some SQLite internals to be around, like certainPRAGMA ...functions,
or the shape of theEXPLAINoutput. We work around this by detecting those
queries and telling bald-faced lies to Datasette.Datasette expectsjson_type(...)to throw asqlite3.OperationalErroron invalid
JSON, but DuckDB will (of course) throw its own type:duckdb.InvalidInputExceptionDuckDB is missing some functions from SQLite:json_each(...),date(...)rowidcolumns in SQLite are stable identifiers. This is not true in DuckDB.SQLite's Python interface supports interrupting long-running queries. DuckDB's
C API supports this, too, but it has not yet been exposed to the Python API.
Seehttps://github.com/duckdb/duckdb/issues/5938andhttps://github.com/duckdb/duckdb/pull/3749Datasette's CustomJSONEncoder only expects objects of the sort that SQLite can
store. DuckDB has native support for thedatetype, which requires patching.DevelopmentTo set up this plugin locally, first checkout the code. Then create a new virtual environment:cd datasette-parquet
python3 -m venv venv
source venv/bin/activateNow install the dependencies and test dependencies:pip install -e '.[test]'To run the tests:pytest