Package: atom-db
atom-db is a database abstraction layer for theatomframework. This package provides api's for
seamlessly saving and restoring atom objects from json based document databases
and SQL databases supported by sqlalchemy.The main reason for building this is to make it easier have database integration
withenamlapplications so a separate
framework is not needed to define database models.This was originally a part ofenaml-webbut has been pulled out to a separate package.OverviewSupports MySQL and PostgresUses django like queries or raw sqlalchemy queriesWorks with alembic database migrationsSupports MongoDB using motorStructureThe design is based somewhat on django.There is a "manager" calledModel.objectsto do queries on the database table
created for each subclass.Serialization and deserialization is done withModel.serializer.Note: As of 0.3.11 serialization can be customizer per member by tagging the
member with aflattenorunflattenwhich should be a async callable which
accepts the value and scope.EachModelhas asyncsave,delete, andrestoremethods to interact with
the database. This can be customized if needed using__restorestate__and__getstate__.MySQL and Postgres supportYou can use atom-db to save and restore atom subclasses to MySQL and Postgres.Just define models using atom members, but subclass theSQLModeland atom-db
will convert the builtin atom members of your model to sqlalchemy table columns
and create asqlalchemy.Tablefor your model.Customizing table creationTo customize how table columns are created you can tag members with information
needed for sqlalchemy columns, exStr().tag(length=40)will make asa.String(40).
Seehttps://docs.sqlalchemy.org/en/latest/core/type_basics.html. Tagging any
member withstore=Falsewill make the member be excluded from the db.atomdb will attempt to determine the proper column type, but if you need more
control, you can tag the member to specify the column type withtype=sa.<type>or specify the full column definition withcolumn=sa.Column(...).If you have a custom member, you can define adef get_column(self, model)ordef get_column_type(self, model)method to create the table column for the
given model.Primary keysYou can tag a member withprimary_key=Trueto make it the pk. If no member
is tagged withprimary_keyit will create and use_idas the primary key.
The_idmember will be always alias to the actual primary key. Use the__pk__attribute of the class to get the name of the primary key member.Table metadataLike in Django a nestedMetaclass  can be added to specify thedb_name,unique_together,composite indexesandconstraints.If nodb_nameis specified on a Meta class, the table name defaults the what
is set in the__model__member. This defaults to the qualname of the class,
egmyapp.SomeModel.classSomeModel(SQLModel):# ...classMeta:db_table='custom_table_name'composite indexesmust be a list of each composite index description. Seesqlachemy's Indexfor index tuple description.First element is the index name. IfNone, the name will be auto-generated according the convention. Followings elements are columns table. Order of columns matters.classSomeModel(SQLModel):# ...classMeta:composite_indexes=[(None,'a','b'),('ìndex_b_c','b','c')]In the exemple above, two composite indexes are created. The first is on columns 'a' and 'b' with nameix_SomeModel_a_b. The second one on columns 'b' and 'c' with nameìndex_b_c.Table creation / droppingOnce your tables are defined as atom models, create and drop tables usingcreate_tableanddrop_tableofModel.objectsrespectively For example:fromatomdb.sqlimportSQLModel,SQLModelManager# Call create_tables to create sqlalchemy tables. This does NOT write them to# the db but ensures that all ForeignKey relations are createdmgr=SQLModelManager.instance()mgr.create_tables()# Now actually drop/create for each of your models# Drop the table for this model (will raise sqlalchemy's error if it doesn't exist)awaitUser.objects.drop_table()# Create the user tableawaitUser.objects.create_table()Themgr.create_tables()method will create the sqlalchemy tables for each
imported SQLModel subclass (anything in the manager'sregistrydict). This
should be called after all of your models are imported so sqlalchemy can
properly setup any foreign key relations.The manager also has ametadatamember which holds thesqlalchemy.MetaDataneeded for migrations.Once the tables are created, they are accessible viaModel.objects.table.Note: The sqlachemy table is also assigned to the__table__attribute of
each model class, however this will not be defined until the manager has
created it.Database setupBefore accessing the DB you must assign a "database engine" to the manager'sdatabasemember.Note: As of0.6.2you can also specify this as a dictionary to use multiple
databases.importosimportrefromaiomysql.saimportcreate_enginefromatomdb.sqlimportSQLModelManagerDATABASE_URL=os.environ.get('MYSQL_URL')# Parse the DB urlm=re.match(r'mysql://(.+):(.*)@(.+):(\d+)/(.+)',DATABASE_URL)user,pwd,host,port,db=m.groups()# Create the engineengine=awaitcreate_engine(db=db,user=user,password=pwd,host=host,port=port)# Assign it to the managermgr=SQLModelManager.instance()mgr.database=engineThis engine will then be used by the manager to execute queries.  You can
retrieve the database engine from any Model by usingModel.objects.engine.Multiple databaseIf you need to use more than one database it looks like this.# Multiple databasesmgr=SQLModelManager.instance()mgr.database={'default':awaitcreate_engine(**default_db_params),'other':awaitcreate_engine(**other_db_params),}To specify which database is used either using the__database__class field
or specify it as thedb_nameon the model Meta.classExternalData(SQLModel):# ... fieldsclassMeta:db_name='other'db_table='external_data'Django style queriesOnly very basic ORM style queries are implemented for common use cases. These
areget,get_or_create,filter, andall. These all accept
"django style" queries using<name>=<value>or<name>__<op>=<value>.For example:john,created=awaitUser.objects.get_or_create(name="John Doe",email="jon@example.com",age=21,active=True)assertcreatedjane,created=awaitUser.objects.get_or_create(name="Jane Doe",email="jane@example.com",age=48,active=False,rating=10.0)assertcreated# Startswithu=awaitUser.objects.get(name__startswith="John")assertu.name==john.name# In queryusers=awaitUser.objects.filter(name__in=[john.name,jane.name])assertlen(users)==2# Is queryusers=awaitUser.objects.filter(active__is=False)assertlen(users)==1andusers[0].active==FalseSeesqlachemy's ColumnElementfor which queries can be used in this way.  Also the tests check that these
actually work as intended.Note: As of0.4.0you can pass sqlalchemy filters as non-keyword arguments
directly to the filter method.Caching, select related, and prefetch relatedForeign key relations can automatically be loaded usingselect_relatedandprefetch_related. Select related will perform a
join while prefetch related does a separate query.Each Model has a cache available atModel.objects.cachewhich uses weakrefs to
ensure the same object is returned each time. You can manually prefetch objects
and atom-db will pull them from it's internal cache when restoring objects.For example with a simple many to one relationship like this:classCategory(SQLModel):name=Str()products=Relation(lambda:Product)classProduct(SQLModel):title=Str()category=Typed(Category)category=awaitCategory.objects.create(name="PCB")awaitProduct.objects.create(title="Stepper driver",category=category)Use select related to load the product's category foreign key automatically.# In this case the category of each product will automatically be loadedproducts=awaitProduct.objects.select_related('category').filter(title__icontains="driver")# The __restored__ flag can be used check if the model has been loadedassertproducts[0].category.name=="PCB"If a foreign key relation is NOT in the cache or in the state from a joined row
it will create an "unloaded" model with only the primary key populated. In this
case the__restored__flag will be set toFalse.From the other direction use prefetch related.category=awaitCategory.objects.prefetch_related('products').get(name="PCB")assertcategory.products[0].title=="Stepper driver"Note: prefetch_related does not apply a limit. If the query has a lot of rows
this may be a problem.Alternatively you can prefetch the related objects and they will be
automatically pulled from the internal cache (egTheModel.objects.cache).all_categories=awaitCategory.objects.all()products=awaitProduct.objects.filter(title__icontains="driver")assertproducts[0].categoryinall_categoriesAdvanced / raw sqlalchemy queriesFor more advanced queries using joins, etc.. you must build the query with
sqlalchemy then execute it. Thesa.Tablefor an atom model can be retrieved
usingModel.objects.tableon which you can use select, where, etc... to build
up whatever query you need.Then usefetchall,fetchone,fetchmany, orexecuteto do these queries.These methods do NOT return an object but the row from the database so they
must manually be restored.When joining you'll usually want to passuse_labels=True.  For example:q=Job.objects.table.join(JobRole.objects.table).select(use_labels=True)forrowinawaitJob.objects.fetchall(q):# Restore each manually, it handles pulling out the fields that are it's ownjob=awaitJob.restore(row)role=awaitJobRole.restore(row)Depending on the relationships, you may need to then post-process these so they
can be accessed in a more pythonic way. This is trade off between complexity
and ease of use.Connections and TransactionsA connection can be retrieved usingModel.objects.connection()and used
like normal aiomysql / aiopg connection. A transaction is done in the same way
as defined in the docs for those libraries eg.asyncwithJob.objects.connection()asconn:trans=awaitconn.begin()try:# Do your queries here and pass the `connection` to eachjob,created=awaitJob.objects.get_or_create(connection=conn,**state)except:awaittrans.rollback()raiseelse:awaittrans.commit()When using a transaction you need to pass the active connection to
each call or it will use a different connection outside of the transaction!The connection argument is removed from the filters/state. If your model happens
to have a member namedconnectionyou can rename the connection argument by
withModel.object.connection_kwarg = 'connection_'or whatever name you like.MigrationsMigrations work usingalembic. The metadata needed
to autogenerate migrations can be retrieved fromSQLModelManager.instance().metadataso add the following
in your alembic's env.py:# Import your db models firstfrommyapp.modelsimport*fromatomdb.sqlimportSQLModelManagermanager=SQLModelManager.instance()manager.create_tables()# Create sa tablestarget_metadata=manager.metadataThe rest is handled by alembic.Note: As of 0.4.1 the constraint naming conventions can be set using
manager.constraints, this must be done before any tables are imported.NoSQL supportYou can also use atom-db to save and restore atom subclasses to MongoDB.The NoSQL version is very basic as mongo is much more relaxed. No restriction
is imposed on what type of manager is used, leaving that to whichever database
library is preferred but it's tested (and currently used) withmotorandtornado.Just define models using atom members, but subclass theNoSQLModel.fromatom.apiimportUnicode,Int,Instance,Listfromatomdb.nosqlimportNoSQLModel,NoSQLModelManagerfrommotor.motor_asyncioimportAsyncIOMotorClient# Set DBclient=AsyncIOMotorClient()mgr=NoSQLModelManager.instance()mgr.database=client.test_dbclassGroup(NoSQLModel):name=Unicode()classUser(NoSQLModel):name=Unicode()age=Int()groups=List(Group)Then we can create an instance and save it. It will perform an upsert or replace
the existing entry.admins=Group(name="Admins")awaitadmins.save()# It will save admins using it's ObjectIDbob=User(name="Bob",age=32,groups=[admins])awaitbob.save()tom=User(name="Tom",age=34,groups=[admins])awaittom.save()To fetch from the DB each model has aModelManagercalledobjectsthat will
simply return the collection for the model type. For example.# Fetch from db, you can use any MongoDB queries herestate=awaitUser.objects.find_one({'name':"James"})ifstate:james=awaitUser.restore(state)# etc...Restoring is async because it will automatically fetch any related objects
(ex the groups in this case). It saves objects using the ObjectID when present.And finally you can either delete using queries on the manager directly or
call on the object.awaittom.delete()assertnotawaitUser.objects.find_one({'name':"Tom"})You can exclude members from being saved to the DB by tagging them
with.tag(store=False).ContributingThis is currently used in a few projects but not considered mature by
any means.Pull requests and feature requests are welcome!