Package: s3sqlite
s3sqliteQuery SQLite databases in S3 using s3fsAPSWSQLite VFS. This VFS enables reading
databases from S3 usings3fs. This only supports
reading operations, any operation that tries to modify the DB file is ignored.Inspired bysqlite-s3vfsandsqlite-s3-query.Notes about journal modeThis VFS will only work when the DB file is in any journal mode that isnotWAL. However, it will work if you set the journal
mode to something else just before uploading the file to S3. You can (and
probably should) use WAL mode to generate the DB. Then you can change the
journal mode (and the page size if you neeed) before uploading it to S3.The test suiteincludestests for that use case. Take into account that the page size can't be changed
when the database is in WAL mode. You need to change it before setting the WAL
mode or by setting the database to rollback journal mode.You need to executeVACUUM;after changing the page
sizein a SQLite database.Example usageimports3fsimports3sqliteimportapsw# Create an S3 filesystem. Check the s3fs docs for more examples:# https://s3fs.readthedocs.io/en/latest/s3=s3fs.S3FileSystem(key="somekey",secret="secret",client_kwargs={"endpoint_url":"http://..."},)s3vfs=s3sqlite.S3VFS(name="s3-vfs",fs=s3)# Define the S3 locationkey_prefix="mybucket/awesome.sqlite3"# Upload the file to S3s3vfs.upload_file("awesome.sqlite3",dest=key_prefix)# Create a database and query itwithapsw.Connection(key_prefix,vfs=s3vfs.name,flags=apsw.SQLITE_OPEN_READONLY)asconn:cursor=conn.execute("...")print(cursor.fetchall())Installationpython3 -m pip install s3sqliteRun testsThe testing script will use theChinook
database, it will modify (andVACUUM;) the file to use all the possible combinations of journal modes and
page sizesDownload the chinook database:curlhttps://raw.githubusercontent.com/lerocha/chinook-database/master/ChinookDatabase/DataSources/Chinook_Sqlite_AutoIncrementPKs.sqlite-ochinook.sqlite3Make sure you have Docker installed.The testing script will start aMinIOcontainer to run the
tests locally. After the tests finish, the container will be stopped
atuomatically.Run the tests:python3-mpytesttest.pyAlternativessqlite-s3vfs: This VFS stores the
SQLite file as separate DB pages. This enables having a single writer without
having to overwrite the whole file.s3sqlite's main difference is that this
just needs uploading a single file to S3.sqlite-s3vfswill split the
database in pages and upload the pages separately to a bucket prefix. Having
just a single file has some advantages, like making use of objectversioning
in the
bucket.
I also think that relying ons3fsmakes the VFS moreflexiblethan callingboto3assqlite3-s3vfsdoes.sqlite-s3-query: This VFS is very
similar tos3sqlit, but it usesctypesdirectly to create the VFS and useshttpxto make requests to S3.I decided to create a new VFS that didn't require usingctypesso that it's
easier to understand and maintain, but I still want to have a single file in S3
(vs. separate DB pages). At the same time, by usings3f3I know I can use any S3
storage supported by that library.OtherThe Chinook database used for testing can be obtained from:https://github.com/lerocha/chinook-database/The testing section in this README contains a command you can run to get the file.LicenseDistributed under the Apache 2.0 license. SeeLICENSEfor more information.