Package: asyncpg-lostream
asyncpg-lostreamCRUD on PostgreSQL large objects using async drivers and asyncio. It is tested against SQLAlchemy.PurposeSynchronous drivers such aspsycopg2support large objects via a file-like interface utilizing libpg. Unfortunately, async drivers such asasyncpgdo not support large objects at all (at this time). This module is designed to provide an interface for PostgreSQL large objects that is compatible with asyncio and async database drivers. This interface is achieved by calling the supporting PostgreSQL stored functions that are an interface to the large object store.This interface is designed to operate on data by reading and writing in chunks so as to stream the data to and from the large object store.This codebase is not tied or affiliated with asyncpg. It does utilize SQLAlchemy's AsyncConnection class in its typing.PostgreSQL Large ObjectsA large object is a special store in PostgreSQL that operates similarly to a file. The large object itself is the record. The database data type isbytea(bytesin Python). The interface operates as a read and write to the allocated large object via read and write functions. (There are more, but they are out of scope for this project.) The data type of the large object id isoid. The tables used by PostgreSQL arepg_largeobjectandpg_largeobject_metadata. Thepg_largeobjecttable holds the data itself. Thepg_largeobject_metadatahas a link to the owner oid. These two tables are linked by theoidreference.When assiciating the large object to a table record, add anoidtype column to hold the allocated large objectoidvalue from a created large object.See the PostgreSQL documentationhere.UtilizationExplicit Createfromasyncpg_lostream.lostreamimportPGLargeObject# It is the responsibility of the caller to resolve how an# AsyncEngine is created and how an AsyncConnection is created.# Create a large objectlob_oid=awaitPGLargeObject.create_large_object(db)# Open a large object for read and writepgl=PGLargeObject(async_connection,lob_oid,mode="rw")withopen("my_data.txt","r")asdata:forbuffindata.read(pgl.chunk_size):awaitpgl.write(buff.encode)pgl.close()Context Manager Createfromasyncpg_lostream.lostreamimportPGLargeObject# It is the responsibility of the caller to resolve how an# AsyncEngine is created and how an AsyncConnection is created.withopen("my_data.txt","r")asdata:asyncwithPGLargeObject(async_connection,0,"w")aspgl:forbuffindata.read(pgl.chunk_size):awaitpgl.write(buff.encode())Context manager readfromasyncpg_lostream.lostreamimportPGLargeObject# It is the responsibility of the caller to resolve how an# AsyncEngine is created and how an AsyncConnection is created.asyncwithPGLargeObject(async_connection,existing_lob_oid,"r")aspgl:asyncforbuffinpgl:print(buff.decode())DevelopmentEnvironmentCreate a virtual environmentpython-mvenv/path/to/venvActivate the virtual environmentsource/path/to/venv/bin/activateEnsure pip is up to datepipinstall--upgradepipInstall packages fromrequirements.txtpipinstall-rrequirements.txtInstallpre-configpre-configinstallDevelopmentmake,docker-composeanddockerare required for development.To list the make targets, usemake helpTo start a local PostgreSQL 13 container:make start_dbTo shutdown the local PostgreSQL 13 container:make stop_dbAfter making changes, create your unit tests in theasyncpg-lostream/testsdirectory.Test your changes with the commandmake testPackagingIf you intend to make a releaase, change the version in thesetup.cfgfile. This value will be copied to the module's__version__file.Build the package usingmake build. This will run the tests and then build the artifacts. These will be put into thedist/directory.For instructions on upload to PyPI, see thePackaging Python Projects Dcoumentation