Package: pyramid-sqlassist
pyramid_sqlassistSQLAssist offers a streamlined integration for handling multipleSQLAlchemydatabase connections
underPyramid.  It ships with first-party
support for "reader", "writer" and "logger" concepts - but can be extended to
support any design.SQLAssist also offers some utility mixin/base classes for SQLAlchemy
applications that are useful for debugging applications. A debugtoolbar panel
is included as well, which can be used to further aid in development.This package has been working in production environments for many years.PRs are always appreciated.Recent changes:Withv0.13.0, SQLAlchemy 1.3.0 and zope.sqlalchemy 1.2.0 are required.Withv0.12.0, there have been some API changes and the introduction of apyramid_debugtoolbarpanel.WARNINGThis package uses scoped Sessions by default.v0.9.1introduced a capability to use non-scoped Sessions.
This appears to work, but has not been tested as thoroughly as I'd like.Non-scoped Sessions are not integrated with thetransactionpackage, as they
were incompatible with Zope's transaction extension when support was last
attempted.There is probably a way to get this to work, or things may have changed.
Pull Requests are welcome.OverviewThe package facilitates managing multiple SQLAlchemy connections under Pyramid
through a single API.  It has been used in Celery too.There are 4 steps to using this package:It is the job of your Pyramid application'smodelto create
SQLAlchemy engines.Each created engine should be passed intopyramid_sqlassist.initialize_engineAfter initializing all the engines, invokepyramid_sqlassist.register_request_methodwith the name of the request
attribute you wish to useSQLAlchemy classes in your model must inherit frompyramid_sqlassist.DeclaredTable-- which is just an instance of
SQLAlchemy'sdeclarative_base.Note: If your Pyramid application connects to the database BEFORE a process
fork, you must callpyramid_sqlassist.reinit_engine(/engine/).  This can be
streamlined with thepyramid_forksafeplugin.What does all this accomplish?pyramid_sqlassistmaintains a private Python dict in it's
namespace:_ENGINE_REGISTRY.Callinginitialize_enginewill wrap each SQLAlchemy engine into a SQLAssistEngineWrapperand then register it into the_ENGINE_REGISTRY.  The wrapper
contains a SQLAlchemysessionmakercreated for each engine, along with some
convenience functions.Callingregister_request_methodwill invoke Pyramid'sadd_request_methodto
add aDbSessionsContaineronto the Pyramid Request as a specified attribute
name.TheDbSessionsContainerautomatically register a cleanup function via
Pyramid'sadd_finished_callbackif the database is used.As a convenience, the active Pyramid request is stashed in each SQLAlchmey
session's "info" dict.ExampleThis is an examplemodel.pyfor a Pyramid app, which creates a READER and
WRITER connection.# model.py
import sqlalchemy
import pyramid_sqlassist

from . import model_objects


def initialize_database(config, settings):

	# setup the reader
	engine_reader = sqlalchemy.engine_from_config(settings,
												  prefix="sqlalchemy_reader.",
												  )
	pyramid_sqlassist.initialize_engine('reader',
										engine_reader,
										is_default=False,
										model_package=model_objects,
										use_zope=False,
										is_scoped=is_scoped,
										)

	# setup the writer
	engine_writer = sqlalchemy.engine_from_config(settings,
												  prefix="sqlalchemy_writer.",
												  echo=sqlalchemy_echo,
												  )
	pyramid_sqlassist.initialize_engine('writer',
										engine_writer,
										is_default=False,
										model_package=model_objects,
										use_zope=False,
										is_scoped=is_scoped,
										)

	# setup the shared interface
	pyramid_sqlassist.register_request_method(config, 'dbSession')Miscellaneous infoBecause Pyramid will lazily create the request database interaction object, it
is very lightweight.  On initialization, the container will register a
cleanup routine viaadd_finished_callback.TheDbSessionsContainerexposes some methods:reader- property. memoized access to "reader" connectionwriter- property. memoized access to "writer" connectionlogger- property. memoized access to "logger" connectionany- property. invokesget_any()get_reader- method. lazy access to "reader" connectionget_writer- method. lazy access to "writer" connectionget_logger- method. lazy access to "logger" connectionget_any- method. tries to find memoized connections.
otherwise will invoke another method.On first access of every "Session", the container will re-initialize that
Session by invoking it as a callable, issuing a.rollback(), and stashing the
current Pyramid request in the Session'sinfodict.Within your code, the request can be retrieved viaobject_sessionfrom sqlalchemy.orm.session import object_session
_session = object_session(ExampleObject)
request = _session.info['request']The cleanup function will callsession.remove()for all Sessions that were
used within the request.A postfork hook is available if needed viareinit_engine.
For all managed engines,engine.dispose()will be called.Why it works:DeclaredTableis simply an instance ofsqlalchemy.ext.declarative.declarative_base, bound to our own metadata# via Pyramid
# Recommended naming convention used by Alembic, as various different database
# providers will autogenerate vastly different names making migrations more
# difficult. See: http://alembic.zzzcomputing.com/en/latest/naming.html
NAMING_CONVENTION = {
	"ix": 'ix_%(column_0_label)s',
	"uq": "uq_%(table_name)s_%(column_0_name)s",
	"ck": "ck_%(table_name)s_%(constraint_name)s",
	"fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s",
	"pk": "pk_%(table_name)s"
}

# store the metadata in the package (GLOBAL)
_metadata = sqlalchemy.MetaData(naming_convention=NAMING_CONVENTION)

# this is used for inheritance only
DeclaredTable = declarative_base(metadata=_metadata)Subclassing tables fromDeclaredTabletakes care of all the core ORM setup.Wheninitialize_engineis called, by defaultsqlalchemy.orm.configure_mappersis triggered (this can be deferred to first
usage of the ORM, but most people will want to take the performance hit on
startup and try to push the mapped tables into shared memory before a fork).Misc Objectsobjects.UtilityObjectcore object with utility methods for quick prototyping of applications.toolsthis namepace is currently unused;
it houses some in-progress code for supporting table reflectiondebugtoolbar supportSimply addpyramid_sqlassist.debugtoolbartodebugtoolbar.includesfor your application.For example, this is a line from a.inifiledebugtoolbar.includes = pyramid_sqlassist.debugtoolbarThe SQLAssist debugtoolbar panel includes information such as:the request attribute theDbSessionsContainerhas been memoized intothe connections which were active for the requestthe engine status for the active request (initialized, started, ended)the engines configured for the application and available to the requestThe panel is intended to help debug issues with connections - though there
should be none.TODO:debugtoolbar: show id information for the underlying connections and
connection pool to help troubleshoot potential forking issuesNotesPYTHONOPTIMIZE. All logging functions are nested underif __debug__:statements; they can be compiled away during productionThanksSections of this code were originally taken from or inspired by:SQLAlchemy docsUsing Thread-Local Scope with Web ApplicationsSession Frequently Asked QuestionsMike Bayer's blog post 'Django-style Database Routers in SQLAlchemy'Pyramid's@reifydecorator andset_request_propertyattributeMike Orr's package 'sqlahelper'this was originally based on FindMeOn™'s Pylons based library "opensocialnetwork"Example UsageIn yourenv.ini, specify multiple sqlalchemy urls (which might be to
different dbs or the same db but with different permissions):sqlalchemy_reader.url = postgres://myapp_reader:myapp@localhost/myapp
sqlalchemy_writer.url = postgres://myapp_writer:myapp@localhost/myapp/init.py:mainfrom . import models

try:
    import uwsgi

    def post_fork_hook():
        models.database_postfork()

    uwsgi.post_fork_hook = post_fork_hook

except ImportError:
    pass

def main(global_config, **settings):
	...
	models.initialize_database(settings)
	.../models/init.pyimport sqlassist

ENGINES_ENABLED = ['reader', 'writer', ]

def initialize_database(settings):

	engine_reader = sqlalchemy.engine_from_config(settings, prefix="sqlalchemy_reader.")
	sqlassist.initialize_engine('reader',engine_reader,default=True, reflect=myapp.models, use_zope=False)

	engine_writer = sqlalchemy.engine_from_config(settings, prefix="sqlalchemy_writer.")
	sqlassist.initialize_engine('writer',engine_writer,default=False, reflect=myapp.models, use_zope=True)

	# custom property: `request.dbSession`
	config.add_request_method(
		request_setup_dbSession,
		'dbSession',
		reify=True,
	)

def database_postfork():
	for i in ENGINES_ENABLED:
		sqlassist.reinit_engine(i)

def request_setup_dbSession(request):
	return sqlassist.DbSessionsContainer(request)/models/actual_models.pyimport sqlalchemy as sa
from sqlassist import DeclaredTable

class TestTable(DeclaredTable):
	__tablename__ = 'groups'

	id = sa.Column(sa.Integer, primary_key=True)
	name = sa.Column(sa.Unicode(255), nullable=False)
	description = sa.Column(sa.Text, nullable=False)in your handlers, you have the following. Note,sqlalchemyis only imported
to grab an exception:import sqlalchemy

class BaseHandler(object):
	def __init__(self,request):
		self.request = request

class ViewHandler(BaseHandler):

	def index(self):

		print self.request.dbSession.reader.query(models.actual_models.TestTable).all()

		try:
			#this should fail , assuming reader can't write
			dbTestTable = models.actual_models.TestTable()
			dbTestTable.name= 'Test Case 1'
			self.request.dbSession.reader.add(dbTestTable)
			self.request.dbSession.reader.commit()

		except sqlalchemy.exc.ProgrammingError:
			self.request.dbSession.reader.rollback()
			raise ValueError("Commit Failed!")

		#but this should work , assuming writer can write
		dbTestTable = models.actual_models.TestTable()
		dbTestTable.name = 'Test Case 2'
		self.request.dbSession.writer.add(dbTestTable)
		self.request.dbSession.writer.commit()UtilityObjectIf you inherit from this class, your SQLAlchemy objects have some convenience
methods:get__by__id( self, dbSession, id , id_column='id' ):get__by__column__lower( self, dbSession, column_name , search , allow_many=False ):get__by__column__similar( self, dbSession , column_name , seed , prefix_only=True):get__by__column__exact_then_ilike( self, dbSession, column_name, seed ):get__range( self, dbSession, start=0, limit=None, sort_direction='asc', order_col=None, order_case_sensitive=True, filters=[], debug_query=False):columns_as_dict(self):Another important note...DbSessionsContainerThis convenience class ONLY deals with 3 connections right now :readerwriterloggerIf you have more/different names - subclass (or create a patch to deal with
dynamic names!)  I didn't have time for that.The reader and writer classes will start with an automatic rollback;
The logger will not.transactionsupportBy default, the package will try to load the following libraries:import transaction
from zope.sqlalchemy import register as zope_registerThis can be disabled with an environment variableexport SQLASSIST_DISABLE_TRANSACTION=1Caveats$$COMMIT$$if you're using "Zope" & "transaction" modules :you need to calltransaction.commitIMPORTANT remember thatmark_changedexists!if you're not using "Zope" & "transaction" modules :you need to call "dbSession_writer.commit()"Rollbacksyou want to callrollbackon the specific database Sessions to control what
is in each onecatching exceptions if you're trying to support bothtransaction.commit()anddbsession.commit()let's say you do this:try:
	dbSession_writer_1.add(object1)
	dbSession_writer_1.commit()
except AssertionError , e:
	print "Should fail because Zope wants this"

# add to writer
dbSession_writer_2.add(object2)

# commit
transaction.commit()in this event, both object1 and object2 will be committed bytransaction.commit()You must explicitly invoke arollbackafter theAssertionErrorReflected Tablesthis package once supported trying to handle table reflection.
It is being removed unless someone wants to do a better job.