Package: zc.zodbwsgi
WSGI Middleware for Managing ZODB Database ConnectionsThe zc.zodbwsgi provides middleware for managing connections to a ZODB
database. It combines several features into a single middleware
component:database configurationdatabase initializationconnection managementoptional transaction managementoptional request retry on conflict errors (using repoze.retry)optionaly limiting the number of simultaneous database connectionsapplications can take over connection and transaction management on
a case-by-case basis, for example to support the occasional
long-running request.It is designed to work with paste deployment and provides a
“filter_app_factory” entry point, named “main”.A number of configuration options are provided. Option values are
strings.configurationA required ZConfig formatted ZODB database configurationIf multiple databases are defined, they will define a
multi-database. Connections will be to the first defined database.initializerAn optional database initialization function of the formmodule:expressionkeyAn optional name of a WSGI environment key for database connectionsThis defaults to “zodb.connection”.transaction_managementAn optional flag (either “true” or “false”) indicating whether the
middleware manages transactions.Transaction management is enabled by default.transaction_keyAn optional name of a WSGI environment key for transaction managersThis defaults to “transaction.manager”. The key will only be
present if transaction management is enabled.thread_transaction_managerAn option flag (either “true” or “false”) indicating whether the
middleware will use a thread-aware transaction mananger (e.g.,
thread.TransactionManager).Using a thread-aware transaction mananger is convenient if you’re
using a server that always a request in the same thread, such as
servers thaat use thread pools, or that create threads for each
request.If you’re using a server, such as gevent, that handles multiple
requests in the same thread or a server that might handle the same
request in different threads, then you should set this option to
false.Defaults to True.retryAn optional retry countThe default is “3”, indicating that requests will be retried up to
3 times.  Use “0” to disable retries.Note that when retry is not “0”, request bodies will be buffered.demostorage_manage_headerAn optional entry that controls whether the filter will support push/pop
support for the underlying demostorage.If a value is provided, it’ll check for that header in the request. If found
and its value is “push” or “pop” it’ll perform the relevant operation. The
middleware will return a response indicating the action taken _without_
processing the rest of the pipeline.Also note that this only works if the underlying storage is a DemoStorage.max_connectionsMaximum number of simultaneous connections.ContentsWSGI Middleware for Managing ZODB Database ConnectionsBasic usageDatabase initializationDisabling transaction managementSuppressing request retryUsing non-thread-aware (non thread-local) transaction managersdemostorage_manage_headerLimiting the number of connectionsEscaping connection and transaction managementDealing with the occasional long-running requestsChanges1.2.0 (2015-03-08)1.1.0 (2014-04-22)1.0.0 (2013-09-15)0.3.0 (2013-03-07)0.2.1 (2013-03-06)0.2.0 (2013-03-06)0.1.0 (2010-02-16)Basic usageLet’s look at some examples.First we define an demonstration “application” that we can pass to our
factory:import transaction, ZODB.POSException
from sys import stdout

class demo_app:
    def __init__(self, default):
        pass
    def __call__(self, environ, start_response):
        start_response('200 OK', [('content-type', 'text/html')])
        root = environ['zodb.connection'].root()
        path = environ['PATH_INFO']
        if path == '/inc':
            root['x'] = root.get('x', 0) + 1
            if 'transaction.manager' in environ:
                environ['transaction.manager'].get().note('path: %r' % path)
            else:
                transaction.commit() # We have to commit our own!
        elif path == '/conflict':
            print >>stdout, 'Conflict!'
            raise ZODB.POSException.ConflictError
        elif path == "/tm":
            tm = environ["transaction.manager"]
            return ["thread tm: " + str(tm is transaction.manager)]
        return [repr(root)]Now, we’ll define our application factory using a paste deployment
configuration:[app:main]
paste.app_factory = zc.zodbwsgi.tests:demo_app
filter-with = zodb

[filter:zodb]
use = egg:zc.zodbwsgi
configuration =
   <zodb>
     <demostorage>
     </demostorage>
   </zodb>Here, for demonstration purposes, we used an in-memory demo storage.Now, we’ll create an application with paste:>>> import paste.deploy, os
>>> app = paste.deploy.loadapp('config:'+os.path.abspath('paste.ini'))The resulting applications has a database attribute (mainly for
testing) with the created database.
Being newly initialized, the database is empty:>>> conn = app.database.open()
>>> conn.root()
{}Let’s do an “increment” request.>>> import webtest
>>> testapp = webtest.TestApp(app)
>>> testapp.get('/inc')
<200 OK text/html body="{'x': 1}">Now, if we look at the database, we see that there’s now data in the
root object:>>> conn.sync()
>>> conn.root()
{'x': 1}Database initializationWe can supply a database initialization function using the initializer
option.  Let’s define an initialization function:import transaction

def initialize_demo_db(db):
    conn = db.open()
    conn.root()['x'] = 100
    transaction.commit()
    conn.close()and update our paste configuration to use it:[app:main]
paste.app_factory = zc.zodbwsgi.tests:demo_app
filter-with = zodb

[filter:zodb]
use = egg:zc.zodbwsgi
configuration =
   <zodb>
     <demostorage>
     </demostorage>
   </zodb>

initializer = zc.zodbwsgi.tests:initialize_demo_dbNow, when we use the application, we see the impact of the
initializer:>>> app = paste.deploy.loadapp('config:'+os.path.abspath('paste.ini'))
>>> testapp = webtest.TestApp(app)
>>> testapp.get('/inc')
<200 OK text/html body="{'x': 101}">Disabling transaction managementSometimes, you may not want the middleware to control transactions.
You might do this if your application used multiple databases,
including non-ZODB databases[1].  You can suppress
transaction management by supplying a value of “false” for the
transaction_management option:[app:main]
paste.app_factory = zc.zodbwsgi.tests:demo_app
filter-with = zodb

[filter:zodb]
use = egg:zc.zodbwsgi
configuration =
   <zodb>
     <demostorage>
     </demostorage>
   </zodb>

initializer = zc.zodbwsgi.tests:initialize_demo_db
transaction_management = falseSuppressing request retryBy default, zc.zodbwsgi addsrepoze.retrymiddleware to retry requests
when there are conflict errors:>>> import ZODB.POSException
>>> app = paste.deploy.loadapp('config:'+os.path.abspath('paste.ini'))
>>> testapp = webtest.TestApp(app)
>>> try: testapp.get('/conflict')
... except ZODB.POSException.ConflictError: pass
... else: print 'oops'
Conflict!
Conflict!
Conflict!
Conflict!Here we can see that the request was retried 3 times.We can suppress this by supplying a value of “0” for the retry option:[app:main]
paste.app_factory = zc.zodbwsgi.tests:demo_app
filter-with = zodb

[filter:zodb]
use = egg:zc.zodbwsgi
configuration =
   <zodb>
     <demostorage>
     </demostorage>
   </zodb>

retry = 0Now, if we run the app, the request won’t be retried:>>> app = paste.deploy.loadapp('config:'+os.path.abspath('paste.ini'))
>>> testapp = webtest.TestApp(app)
>>> try: testapp.get('/conflict')
... except ZODB.POSException.ConflictError: pass
... else: print 'oops'
Conflict!Using non-thread-aware (non thread-local) transaction managersBy default, the middleware uses a thread-aware transaction manager:[app:main]
paste.app_factory = zc.zodbwsgi.tests:demo_app
filter-with = zodb

[filter:zodb]
use = egg:zc.zodbwsgi
configuration =
   <zodb>
     <demostorage>
     </demostorage>
   </zodb>
initializer = zc.zodbwsgi.tests:initialize_demo_dbThis can be controlled via thethread_transaction_managerkey:[app:main]
paste.app_factory = zc.zodbwsgi.tests:demo_app
filter-with = zodb

[filter:zodb]
use = egg:zc.zodbwsgi
configuration =
   <zodb>
     <demostorage>
     </demostorage>
   </zodb>
initializer = zc.zodbwsgi.tests:initialize_demo_db
thread_transaction_manager = falsedemostorage_manage_headerProviding an value for this options enables hooks that allow one to push/pop
the underlying demostorage.[app:main]
paste.app_factory = zc.zodbwsgi.tests:demo_app
filter-with = zodb

[filter:zodb]
use = egg:zc.zodbwsgi
configuration =
   <zodb>
     <demostorage>
     </demostorage>
   </zodb>

key = connection
transaction_key = manager
demostorage_manage_header = X-FOOIf the push or pop header is provided, the middleware returns a response
immediately without sending it to the end of the pipeline.>>> testapp.get('/', {}, headers={'X-FOO': 'push'}).body
'Demostorage pushed\n'>>> testapp.get('/inc')
<200 OK text/html body="{'x': 2}">>>> testapp.get('/', {}, {'X-FOO': 'pop'}).body
'Demostorage popped\n'>>> testapp.get('/')
<200 OK text/html body="{'x': 1}">If you have access to the middleware object, you can accomplish the
same thing by calling the push and pop methods, which also return the
database. This is useful when you’re running the server in the test
process and have Python access:>>> db = app.application.push()Note that app is a repoze.retry, so we have to use .application to get
the wsgi app.>>> with db.transaction() as conn:
...     conn.root.x = 41>>> testapp.get('/inc')
<200 OK text/html body="{'x': 42}">>>> db = app.application.pop()
>>> with db.transaction() as conn:
...     print conn.root.x
1>>> testapp.get('/')
<200 OK text/html body="{'x': 1}">This also works with multiple dbs.class demo_app:
    def __init__(self, default):
        pass
    def __call__(self, environ, start_response):
        start_response('200 OK', [('content-type', 'text/html')])
        path = environ['PATH_INFO']
        root_one = environ['connection'].get_connection('one').root()
        root_two = environ['connection'].get_connection('two').root()
        if path == '/inc':
            root_one['x'] = root_one.get('x', 0) + 1
            root_two['y'] = root_two.get('y', 0) + 1
            environ['manager'].get().note('path: %r' % path)

        data = {'one': root_one,
                'two': root_two}

        return [repr(data)][app:main]
paste.app_factory = zc.zodbwsgi.tests:demo_app
filter-with = zodb

[filter:zodb]
use = egg:zc.zodbwsgi
configuration =
   <zodb one>
     <demostorage>
     </demostorage>
   </zodb>
   <zodb two>
     <demostorage>
     </demostorage>
   </zodb>

key = connection
transaction_key = manager
demostorage_manage_header = X-FOOIf the storage of any of the databases is not a demostorage, an error is
returned.[app:main]
paste.app_factory = zc.zodbwsgi.tests:demo_app
filter-with = zodb

[filter:zodb]
use = egg:zc.zodbwsgi
configuration =
   <zodb one>
     <demostorage>
     </demostorage>
   </zodb>
   <zodb two>
     <filestorage>
       path /tmp/Data.fs
     </filestorage>
   </zodb>

key = connection
transaction_key = manager
demostorage_manage_header = fooLimiting the number of connectionsIf you’re using a threaded server, one that dedicates a thread to each
active request, you can limit the number of simultaneous database
connections by specifying the number with themax_connectionsoption.(This only works for threaded servers because it uses threaded
semaphores. In the future, support for other locking mechanisms, such
as gevent Semaphores, may be added. In the mean time, if you’re
inclined to monkey patch, you can replacezc.zodbwsgi.Semaphorewith an alternative semaphore implementation, like gevent’s.)Escaping connection and transaction managementNormally, having connections and transactions managed for you is
convenient. Sometimes, however, you want to take over transaction
management yourself.If you closeenviron['zodb.connection'], then it won’t be closed
byzc.zodbwsgi, nor willzc.zodbwsgicommit or abort the
transaction it started.  If you’re usingmax_connections, closingenviron['zodb.connection']will make the connection available for
other requests immediately, rather than waiting for your request to
complete.Dealing with the occasional long-running requestsDatabase connections can be pretty expensive resources, especially if
they have large database caches.  For this reason, when using large
caches, it’s common to limit the number of application threads, to
limit the number of connections used.  If your application is compute
bound, you generally want to use one application thread per process
and a process per processor on the host machine.If your application itself makes network requests (e.g calling
external service APIs), so it’s network/server bound rather than
compute bound, you should increase the number of application threads
and decrease the size of the connection caches to compensate.If your application is mostly compute bound, but sometimes calls
external services, you can take a hybrid approach:Increase the number of application threads.Setmax_connectionsto 1.In the parts of your application that make external service calls:Closeenviron['zodb.connection'], committing first, if
necessary.Make your service calls.Open and close ZODB connections yourself when you need to use the
database.If you’re using ZEO or relstorage, you might want to create
separate database clients for use in these calls, configured with
smaller caches.Changes1.2.0 (2015-03-08)Record a request summary in transaction meta-dataAdded dependencies to reflect imports.1.1.0 (2014-04-22)Provide Pythonpushandpopmethods for use when testing and
when running the server in the test process.1.0.0 (2013-09-15)Added support for occasional long-running requests:You can limit the number of database connections with
max_connections.You can take over connection and transaction management to release
connections while blocking (typically when calling external
services).Add an option to use a thread-aware transaction manager, and make it
the default.0.3.0 (2013-03-07)Using the demostorage hook now returns a response immediately without
processing the rest of the pipeline. Makes use of this feature less
confusing.0.2.1 (2013-03-06)Fix reference to a file that was renamed.0.2.0 (2013-03-06)Add hooks to manage (push/pop) underlying demostorage based on headers.Refactor filter to use instance attributes instead of a closure.0.1.0 (2010-02-16)Initial release[1]If you want to use multiple ZODB databases, you can
simply define them in your configuration option.  Just make sure to
give them names.  When you want to access a database, use theget_connectionmethod on the connection in the environment:foo_conn = environ['zodb.connection'].get_connection('foo')