Package: django-oraclepool
ILRT Django Oracle poolEd Crewe,IT Services R&Dat University of Bristol, August 2014Packaged version ofhttp://code.djangoproject.com/ticket/7732by Taras Halturin
django database backend that uses cx_Oracle session pooling for connectionsSeehttp://bitbucket.org/edcrewe/django-oraclepoolOriginal Code ModificationsPruned original ticket’s base.py to just hold the pooling relevant code.
Using the standard Oracle connection for the rest of the database classes,
ie. operations, client and introspection.Tested with django 1.1 to 1.6Extra featuresAdded the pooling and logging parameters to the settings.The connector uses the standard python logging model and caters for logging
full details of queries, either to a file log or appending them to the
bottom of the screen if the log level is DEBUG.Added an option for running against existing (older) Oracle databases, ie those
which may not have unicode for character fields.Option also allows running the tests against an existing database so that
running tests doesnt require database creation (oracle sys dba) rights.Added a modification to the cursor to not parse parameters if not required.Why use it?Perhaps due to our remotely distributed Oracle network taking a very long time
to establish connections, the use of cx_Oracle’s session pooling for
connections provided a truely radical performance boost for requests
from 3-4 secs/req to 0.4 secs/req, so many times faster.For single direct Oracle access it might still give a doubling of performance.
Install it and run the performance test to find out (see below).InstallationDownload the egg (or use buildout) or download the tarball and extract it.
Then add /path/to/django-oraclepool to your python path.Specify DATABASE_ENGINE = ‘oraclepool’ instead of ‘oracle’ in your settings.If you dont want to use the default extra database settings then the following defaults
are used>>> EXTRAS = {'min':4,        # starting number of pooled connections
...           'max':8,         # maximum number of connections in the pool
...           'increment':1,   # increase by this amount when more are needed
...           'threaded':True, # server platform optimisation
...           'timeout':600,   # connection timeout, 600 = 10 mins
...           'log':0,         # extra logging functionality turned on
...           'logfile':'',    # file system path to log file
...           'existing':''    # Type modifications for existing database and flag for tests
...           'session':[]     # Add session optimisations applied to each fresh connection, eg.
...                            #   alter session set cursor_sharing = similar;
...                            #   Enables use of bind variables assuming it isnt set at a system level
...                            #   alter session set session_cached_cursors = 20;
...                            #   Allows cursor reuse between queries
...            'like':'LIKE'   # Option instead of LIKEC default which can stop indexes being used
...            }Note that if you want sql logging to screen when in DEBUG mode then add
‘oraclepool.log_sql.SQLLogMiddleware’ to your MIDDLEWARE_CLASSESGeneral PerformanceInitially after apache restart you will see the first few requests taking the same time as each
one initiates a new pooled connection. Then request speed should drop as it loses the Oracle
connection time.NB: Note that if you have PythonDebug On then the pool may get flushed much more regularly.
So you will often get the slower pool populating requests.Using mod_wsgi rather than mod_python may give a 25% added increase although this needs
confirmation on a production instance.It should be remembered that there is also a very great deal of performance work that can be
done at the database level. I have posted a page with some of the tips and tricks for
improving database performance on my blog -http://edcrewe.blogspot.co.uk/p/database-performance.htmlPooling alternativesPyora poolAlso tried out pyora pool seehttp://code.google.com/p/pyorapoolbut found increase was only around 90% and also had
issues with connection control and database edits failing.
This also requires the whole architecture of a separate remote procedure call daemon
that holds the connection pool. Uneccesary here … although useful
for pooling across different applications, or multiple servers.ORM poolsUsually ORMs have a generic pooling capability, unfortuately djangos only has a beta one
in development, unless you plugin another ORM, eg.http://www.sqlalchemy.org/instead.
However that does require code rewriting.Having said that a generic ORM level pool is unlikely to perform as well as one at the
database connector level, which in turn is going to be less fast than one within the
database itself (see below).TestsThe tests are run via the normal test command, however in order to test everything OK
the settings file within oraclepool.tests.settings should be used eg.bin/django-admin.py test oraclepool –settings=oraclepool.tests.settingsor the applications adapted to add the oraclepool.tests sub-applications.NB: Some of these tests are derived from a set forhttp://code.google.com/p/django-mssql/They also include the option to run the test suite against an existing database for users who dont
have full oracle dba rights on their test oracle servers.The key extra tests are performance timings for running the test suite via the pooled oracle
connection vs the standard one. Hopefully these timings should indicate whether using oraclepool is
of value when using django with your oracle server network.The performance test simulates a real environment by running up a number of connections
as would exist with a production web server (the Apache2 default is 2 processes * 64 threads)
whilst the test creates a maximum of only 32.
In practise I found the actual performance improvement significantly greater than that indicated
by the doubling of speed that the multiple connections performance test gives. However that
may not be the case dependent on your production oracle and web server environment.SouthIf you use south you need to add the following - see oraclepool.tests.settings:SOUTH_DATABASE_ADAPTERS = { ‘default’: “south.db.oracle” }TODOOracle 11g has internal session pooling - DRCP - and cx_Oracle can use that for
even better performance, so need to add appropriate modification to test whether
the database 11g or later, and use this feature in place of a cx_Oracle side pool.
NB: requires cx_Oracle 5, seehttp://www.oracle.com/technology/pub/articles/tuininga-cx_oracle.htmlChangelog1.4 Test with Django 1.5/1.6 and South - (2014-08-11)Fix transaction wrapper - set autocommit each time add a conn to pool
Note - needs autocommit true on connection to use atomic or else writes not commited consistentlyUse Oracle database wrapper rather than base to keep more in sync
[Ed Crewe]1.3 Add fix for multiple EXTRAs existing - (2013-02-17)Add fix for default value for multiple EXTRAs existing
[Nikita Zubchick]Fix dbshell by adding base._connect_string property
[Bug reporter - skyl]Minor tweaks - e.g. type casting of settings
[Ed Crewe]1.2 Make sure only live connections are used (2012-11-22)Add get_alive_connection wrapper to conn.ping() test connections before use [Brent Watson]Try Oracle rollback if requested [Brent Watson]Pass logger from connection to cursor [Brent Watson]Use retry on connections in case one has died [Ricardo del Cid]Add models.py so test suite runs [Ed Crewe]1.1 Refactor the tests (2012-11-18)Make sure the use existing clause works OK for current database settingsMove tests into oraclepool testsSplit out the tests from the models filesFix the performance test so that its against oracle
(not oraclepool itself, since the install can override the oracle engine class)[Ed Crewe]1.0 Add multiple connection extras configuration (2012-11-17)Use logger for credentials error, if available, and hide pw [Ed Crewe]Fix tests for running against an existing database [Ed Crewe]Fix logger usage so its OK when logging cursor errors [Ed Crewe]Add extras dictionary and logger to connections, rather than globally so each one
can be configured and log separately [Brent Watson]0.9 Django 1.4 compatibility fixes (2012-05-18)Fix database connection usage for django 1.4 compatibility
[Ed Crewe]Fix tests/monkeypatch for performance and regression tests to work OK
[Ed Crewe]Add alias to pool to ensure each connection has its own pool
[Marc Bee]0.8 Django 1.3 compatibility fixes (2011-04-08)Make self.features = DatabaseFeatures(self) to work with oracle connector changeFix the timezone for the testsImprove the error reporting if the connection fails due to wrong credentials etc.[Ed Crewe]0.7 - Oracle LIKEC performance work around and bug fixes (2010-09-20)Optional switch of use of LIKEC for LIKE since LIKEC ignores indexes in old Oracle
so has much worse performance, seehttp://code.djangoproject.com/ticket/11017Tested an application and the test suite with django 1.2.3 and python 2.6
[Ed Crewe]Clean up raising of strings, required for python 2.6 or higher
[Reinout van Rees]Lost an indent along the way for log_sql append to screen
[kylen]Fix multiple pool creation bug under heavy initial load - with locking
[Aa`Koshh]Fix old settings style with django 1.1 python
[akhenakh]0.6 - Minor bugs and django 1.2 compatibility fixes (2010-06-18)Use django 1.2 database settings format by defaultFix a bug with adding sql log to page when in DEBUGRemove print statements where not critical failiure since they will cause one with wsgiDrop homogeneous extra argument since unecessary and breaks if used with earlier cx_OracleFix tests for earlier cx_OracleRemove unused separate class declaration for poolsMake sure all tests passEnsure default settings are picked up when using old format settings with 1.2 [Marc Bee][Ed Crewe]0.5 - Fix logger bug and add session config (2010-02-19)Fix a bug preventing the logger from being usedMake sure session settings are only set on connection creation not cursor useAdd a session list of sql lines for optionally setting session parametersWrap connection signal in try except for older django compatibility.[Ed Crewe]0.4 - Fix performance test (2010-01-29)Whoops, I realised the performance test was underplaying the improvement due to a bug,
it now gives the same performance for single connections (as expected) and 300%
improvement for multiple ones.Added signal for connection ‘creation’ and fixed raise of unicode error[Ed Crewe]0.3 - Initial release (2010-01-28)Added test suite including performance test to demonstrate benefitsTested and made compatible with django 1.2 (as well as django 1.1)Cleaned up the code and factored out anything that wasnt specific to pooling,
subclassing most components from the standard Oracle engine.Modified logging options so DEBUG switches on log to screen and EXTRAS[‘log’]
switches logging to file option separately.[Ed Crewe]0.2 - Used for UoB student repsCommented out debug print statements from DatabaseWrapper._cursor which broke install in wsgiAdded DatabaseWrapper.__init__ like the oracle backend with client, creation and introspection
borrowed from it, since creation is expected in django 1.1.1 code.Added default value of None for settings in DatabaseWrapper._cursor since this argument is not
expected in django 1.1.1 code.Added DATABASE_EXTRAS dictionary to set all the session parameters from django config.Added session pool settings report method, to check these.Added python logger option and sql_log middleware[Ed Crewe]0.1 - UnreleasedInitial release - packaging of code fromhttp://code.djangoproject.com/ticket/7732[Taras Halturin, et al.]