Package: intake-sqlite
SQLite Driver for Intake Data CatalogsThis package provides a (very) thin wrapper around the more generalintake-sqldriver, which can be used to generateIntake
data catalogsfrom SQL databases.Theintake-sqldriver takes anSQL Alchemy database URLand uses it to
connect to and extract data from the database. This works with just fine withSQLite databases, but only when the database
file is stored locally and can be referenced with a simple path.For example this path:/home/zane/code/catalyst/pudl-work/sqlite/pudl.sqlitewould correspond to this SQL Alchemy database URL:sqlite:///home/zane/code/catalyst/pudl-work/sqlite/pudl.sqliteBut you can’t access a remote SQLite DB this way.Why access a remote SQLite DB?SQLite databases are great standalone, standardized containers for relational data,
that can be accessed using a huge variety of tools on almost any computer platform.
They are even accepted as an archival format by the US Library of Congress!Data evolves over time, and it’s often useful to have easy access to several
different versions of it, and to know exactly which version you’re working with.Cloud object storage is extremely cheap and convenient, and makes it easy to
publish and store historical file-based data releases.Managing your own bespoke local filesystem hierarchy filled with data – and
coordinating with colleagues so that everyone is using the same filesystem
organizational scheme – is a pain.Intake catalogs can provide easy access to metadata and let you manage data versions
just like software versions. Installing a new version of the data catalog points you
at the new version of the data.The overhead and cost associated with setting up and maintaining a database that uses
a client-server model is relatively large compared to distributing a few files that
change infrequently, are essentially read-only resources, and only take up a few
gigabytes of space.How does it work?Rather than using an SQL Alchemy database URL to reference the SQLite DB, this intake
driver takes a local path or a remote URL, like:../pudl-work/sqlite/pudl.sqlitehttps://global-power-plants.datasettes.com/global-power-plants.dbs3://cloudy-mc-cloudface-databucket/v1.2.3/mydata.dbFor local paths, it resolves the path and prependssqlite://before handing it off
tointake-sqlto do all the hard work.For remote URLs it usesfsspectocache a local copyof the database, and then givesintake-sqla database URL that points to the cached
copy.importintake_sqlitegpp_cat=intake_sqlite.SQLiteCatalog(urlpath="https://global-power-plants.datasettes.com/global-power-plants.db",storage_options={"simplecache":{"cache_storage":"/home/zane/.cache/intake"}},)list(gpp_cat)# ['global-power-plants',#  'global-power-plants_fts',#  'global-power-plants_fts_config',#  'global-power-plants_fts_data',#  'global-power-plants_fts_docsize',#  'global-power-plants_fts_idx']About Catalyst CooperativeCatalyst Cooperativeis a small group of data
wranglers and policy wonks organized as a worker-owned cooperative consultancy.
Our goal is a more just, livable, and sustainable world. We integrate public
data and perform custom analyses to inform public policy (Hire us!). Our focus is primarily on mitigating
climate change and improving electric utility regulation in the United States.Contact UsFor general support, questions, or other conversations around the project
that might be of interest to others, check out theGitHub DiscussionsIf you’d like to get occasional updates about our projectssign up for our email list.Want to schedule a time to chat with us one-on-one? Join us forOffice HoursFollow us on Twitter:@CatalystCoopMore info on our website:https://catalyst.coopFor private communication about the project or to hire us to provide customized data
extraction and analysis, you can email the maintainers:pudl@catalyst.coop