artemp/trac2github — Issue #141: postgis: ability to handle large resultsets
Author: artemp
Created: 2011-10-09 20:15:59+00:00
State: open
Labels: PostGIS_Input_Driver

If you're querying a massive number of features in mapnik, the entire resultset is transferred in memory from postgis to mapnik, then rendered into the output image. Which can eat lots of RAM and cause Bad Things(TM) to happen.

The attached patch adds to the postgis plugin:
* an optional use of server-side cursors, with a specified number of records to fetch each iteration via the `cursor_size` parameter. <=0 means don't use cursors (default)
* an ability to specify a maximum number of features to return in any query (using `LIMIT`) via an optional `row_limit` parameter. <0=unlimited (default)
* a new `IResultSet` interface, with implementations in `ResultSet` (existing) and `CursorResultSet` (new). Change other code to use `IResultSet` instead of `ResultSet`.

Usage (python):
{{{
mylayer.datasource = mapnik.Datasource(
type = 'postgis',
table = 'test',
host = 'localhost',
dbname = 'test',
user = 'me',
password = 'me',
record_limit = 50000, # limit of 50K features in any query
# against this datasource
cursor_size = 2000 # use server-side cursor, fetch 2K
# features from the DB at a time
)
}}}

'''Note:''' the patch includes the contents of #139 as well.