Package: turnstile
Turnstile is a piece of WSGI middleware that performs true distributed
rate-limiting.  System administrators can run an API on multiple
nodes, then place this middleware in the pipeline prior to the
application.  Turnstile uses a Redis database to track the rate at
which users are hitting the API, and can then apply configured rate
limits, even if each request was made against a different API node.Installing TurnstileTurnstile can be easily installed like many Python packages, usingPIP:pip install turnstileYou can install the dependencies required by Turnstile by issuing the
following command:pip install -r .requiresFrom within your Turnstile source directory.If you would like to run the tests, you can install the additional
test dependencies in the same way:pip install -r .test-requiresNote that the test suite is currently written to work with Python 2.7,
even though Turnstile itself should work with Python 2.6.Adding and Configuring TurnstileTurnstile is intended for use with PasteDeploy-style configuration
files.  It is a filter, and should be placed in an appropriate place
in the WSGI pipeline such that the limit classes used with Turnstile
can access the information necessary to make rate-limiting decisions.
(With theturnstile.limits:Limitclass provided by Turnstile, no
additional information is required, as that class does not
differentiate between users of your application.)The filter section of the PasteDeploy configuration file will also
need to contain enough information to allow Turnstile to access the
database.  Other options may be configured from here as well, such as
thepreprocessconfiguration variable.  The simplest example of
Turnstile configuration would be:[filter:turnstile]
use = egg:turnstile#turnstile
redis.host = <your Redis database host name or IP>The following are the recognized configuration options:configAllows specification of an alternate configuration file.  This can
be used to generate a single file which can be shared by WSGI
servers using the Turnstile middleware and the various provided
tools.  This can also allow for separation of code-related options,
such as thepreprocessoption, from pure configuration, such as
theredis.hostoption.  The configuration file is an
INI-formatted file, with section names corresponding to the first
segment of the configuration option name.  That is, theredis.hostoption would be set as follows:[redis]
host = <your Redis database host name or IP>Configuration options which have no prefix are grouped under the[turnstile]section of the file, as follows:[turnstile]
status = 404 Not FoundNote that specifying theconfigoption in the[turnstile]section will have no effect; it is not possible to cause another
configuration file to be included in this way.control.channelSpecifies the channel that the control daemon listens on.  (See
below for more information about the purpose of the control daemon.)
This option defaults to “control”.control.errors_channelSpecifies the channel that the control daemon (see below) reports
errors to.  This option defaults to “errors”.control.errors_keySpecifies the key of a set in the Redis database to which errors
will be stored.  This option defaults to “errors”.control.limits_keyThe key under which the limits are stored in the database.  See the
section on tools for more information on how to load and dump the
limits stored in the Redis database.  This option defaults to
“limits”.control.node_nameThe name of the node.  If provided, this option allows the
specification of a recognizable name for the node.  Currently, this
node name is only reported when issuing a “ping” command to the
control daemon (see below), and may be used to verify that all hosts
responded to the ping.control.reload_spreadWhen limits are changed in the database, a command is sent to the
control daemon (see below) to cause the limits to be reloaded.  As
having all nodes hit the Redis database simultaneously may overload
the database, this option, if set, allows the reload to be spread
out randomly within a configured interval.  This option should be
set to the size of the desired interval, in seconds.  If not set,
limits will be reloaded immediately by all nodes.control.remoteIf set to “on”, “yes”, “true”, or “1”, Turnstile will connect to a
remote control daemon (see theremote_daemontool described
below).  This enables Turnstile to be compatible with WSGI servers
which use multiple worker processes.  Note that the configuration
valuescontrol.remote.authkey,control.remote.host, andcontrol.remote.portare required.control.remote.authkeySet to an authentication key, for use whencontrol.remoteis
enabled.  Must be the value used by the invocation ofremote_daemon.control.remote.hostSet to a host name or IP address, for use whencontrol.remoteis
enabled.  Must be the value used by the invocation ofremote_daemon.control.remote.portSet to a port number, for use whencontrol.remoteis enabled.
Must be the value used by the invocation ofremote_daemon.control.shard_hintCan be used to set a sharding hint which will be provided to the
listening thread of the control daemon (see below).  This hint is
not used by the default RedisConnectionclass.preprocessContains a list of preprocessor functions, specified as
“module:function” pairs separated by spaces.  During each request,
each preprocessor will be called in turn, with the middleware object
(from which can be obtained the database handle, as well as the
configuration) and the request environment as arguments.  Note that
any exceptions thrown by the preprocessors will not be caught, and
request processing will be halted; this will likely result in a 500
error being returned to the user.redis.connection_poolIdentifies the connection pool class to use.  If not provided,
defaults toredis.ConnectionPool.  This may be used to allow
client-side sharding of the Redis database.redis.connection_pool.connection_classIdentifies the connection class to use.  If not provided, the
appropriateredis.Connectionsubclass for the configured
connection is used (redis.Connectionifredis.hostis
specified, elseredis.UnixDomainSocketConnection).redis.connection_pool.max_connectionsAllows specification of the maximum number of connections to the
Redis database.  Optional.redis.connection_pool.parser_classIdentifies the parser class to use.  Optional.  This is an advanced
feature of theredispackage used by Turnstile.redis.connection_pool.*Any other configuration value provided in theredis.connection_pool.hierarchy will be passed as keyword
arguments to the configured connection pool class.  Note that the
values will be passed as strings.redis.dbIdentifies the specific sub-database of the Redis database to be
used by Turnstile.  If not provided, defaults to 0.redis.hostIdentifies the host name or IP address of the Redis database to
connect to.  Eitherredis.hostorredis.unix_socket_pathmust be provided.redis.passwordIf the Redis database has been configured to use a password, this
option allows that password to be specified.redis.portIdentifies the port the Redis database is listening on.  If not
provided, defaults to 6379.redis.socket_timeoutIf provided, specifies an integer socket timeout for the Redis
database connection.redis.unix_socket_pathNames the UNIX socket on the local host for the local Redis database
to connect to.  Eitherredis.hostorredis.unix_socket_pathmust be provided.statusContains the status code to return if rate limiting is tripped.
This defaults to “413 Request Entity Too Large”.  Note that this
value must start with the 3-digit HTTP code, followed by a space and
the text corresponding to that status code.  Also note that,
regardless of the status code, Turnstile will include theRetry-Afterheader in the response.  (The value of theRetry-Afterheader will be the integer number of seconds until
the request can be retried.)turnstileIf set, identifies an alternate class to use for the Turnstile
middleware.  This can be used in conjunction with subclassingturnstile.middleware:TurnstileMiddleware, which may be done to
override how over-limit conditions are formatted.Other configuration values are available to the preprocessors and theturnstile.limits:Limitsubclasses, but extreme care should be
taken that such configurations remain in sync across the entire
cluster.The Control DaemonTurnstile stores the limits configuration in the Redis database, in
addition to the ephemeral information used to check and enforce the
rate limits.  This makes it possible to change the limits dynamically
from a single, central location.  In order to facilitate such changes,
each Turnstile instance uses an eventlet thread to run a “control
daemon.”  The control daemon uses the publish/subscribe support
provided by Redis to listen for commands, of which two are currently
recognized: ping and reload.Some WSGI servers cannot use Turnstile in this mode, due to using
multiple processes (typically through use of the “multiprocessing”
Python module).  In these circumstances, the control daemon may be
started in its own process (see theremote_daemontool).  Enabling
this requires that thecontrol.remoteconfiguration option be
turned on, and values provided forcontrol.remote.authkey,control.remote.host, andcontrol.remote.port.  See the
documentation for these options for more information.It is possible to configure the listening thread of the control daemon
to use alternate configuration for connecting to the Redis database.
The defaults will be drawn from the[redis]section of the
configuration, but by specifyingredis.*options in the[control]section of the configuration, specific values may be
overridden.The Ping CommandThe “ping” command is the simplest of the control daemon commands.  In
its simplest form, the message “ping:<channel>” is written to the control
channel, which will cause all running Turnstile instances to return
the message “pong” to the specified channel.  If thecontrol.node_nameconfiguration option has been set, this node
name will be included in the response, as “pong:<node name>”.
Finally, additional data (such as a timestamp) can be included in the
“ping” command, as in the message “ping:<channel>:<timestamp>”; this
data will be appended to the response, i.e., “pong:<node
name>:<timestamp>”.  This could be used to verify that all nodes are
responding and not too heavily loaded.(Note that ifcontrol.node_nameis not specified, the response to
a “ping” command containing additional data such as a timestamp will
be “pong::<timestamp>”.)Note that, at present, no tool exists for sending pings or receiving
pongs.The Reload CommandThe “reload” command is the real reason for the existence of the
control daemon.  This command causes the current set of limits to be
reloaded from the database and presented to the middleware for
enforcement.The simplest form of the reload command is simply, “reload”.  If thecontrol.reload_spreadconfiguration option was set, the reload
will be scheduled for some time within the configured time interval;
otherwise, it will be performed immediately.The next simplest form of the reload command is “reload:immediate”.
This causes an immediate reload of the limits, overriding any
configured time spread.The final form of the reload command is “reload:spread:<interval>”,
where the “<interval>” specifies a time interval, in seconds, over
which to spread reloading of the limits.  This specified interval is
used in preference to that specified bycontrol.reload_spread, if
set.Note that thesetup_limitstool automatically initiates a reload
once the limits are updated in the database.  See the section on tools
for more information.Turnstile ToolsThe limits are stored in the Redis database using a sorted set, and
they are encoded using Msgpack.  (Although the Msgpack format is not
human-readable, it is very space and time efficient, which is why it
was chosen for this application.)  This makes manual management of the
limits configuration more difficult, and so Turnstile ships with two
tools to make management of the rate limiting configuration easier.  A
third tool starts up a remote control daemon, for use when Turnstile
is used with applications that run multiple processes, such as thenova-apicomponent of OpenStack.Thedump_limitsToolThedump_limitstool may be used to dump the current limits in the
database into an XML representation.  This tool requires the name of
an INI-style configuration file; see the section on configuring the
tools below for more information.A usage summary fordump_limits:usage: dump_limits [-h] [--debug] config limits_file

Dump the current limits from the Redis database.

positional arguments:
  config       Name of the configuration file, for connecting to the Redis
               database.
  limits_file  Name of the XML file that the limits will be dumped to.

optional arguments:
  -h, --help   show this help message and exit
  --debug, -d  Run the tool in debug mode.Theremote_daemonToolTheremote_daemontool may be used to start a separate control
daemon process.  This tool requires the name of an INI-style
configuration file; see the section on configuring the tools below for
more information.  Note that, in addition to the required Redis
configuration values, configuration values for thecontrol.remote.authkey,control.remote.host, andcontrol.remotes.portoptions must be provided.A usage summary forremote_daemon:usage: remote_daemon [-h] [--log-config LOGGING] [--debug] config

Run the external control daemon.

positional arguments:
  config                Name of the configuration file.

optional arguments:
  -h, --help            show this help message and exit
  --log-config LOGGING, -l LOGGING
                        Specify a logging configuration file.
  --debug, -d           Run the tool in debug mode.Thesetup_limitsToolThesetup_limitstool may be used to read an XML file (such as
that produced bydump_limits) and load the rate limiting
configuration into the Redis database.  This tool requires the name of
an INI-style configuration file; see the section on configuring the
tools below for more information.A usage summary forsetup_limits:usage: setup_limits [-h] [--debug] [--dryrun] [--noreload]
                    [--reload-immediate] [--reload-spread SECS]
                    config limits_file

Set up or update limits in the Redis database.

positional arguments:
  config                Name of the configuration file, for connecting to the
                        Redis database.
  limits_file           Name of the XML file describing the limits to
                        configure.

optional arguments:
  -h, --help            show this help message and exit
  --debug, -d           Run the tool in debug mode.
  --dryrun, --dry_run, --dry-run, -n
                        Perform a dry run; inhibits loading data into the
                        database.
  --noreload, -R        Inhibit issuing a reload command.
  --reload-immediate, -r
                        Cause all nodes to immediately reload the limits
                        configuration.
  --reload-spread SECS, -s SECS
                        Cause all nodes to reload the limits configuration
                        over the specified number of seconds.Configuring the ToolsThe toolsdump_limits,remote_daemon, andsetup_limitsrequire an INI-style configuration file, which specifies how to
connect to the Redis database.  This file should contain the section
“[redis]” and should be populated with the same “redis.*” options as
the PasteDeploy configuration file, minus the “redis.” prefix.  For
example:[redis]
host = <your Redis database host name or IP>Each “redis.*” option recognized by the Turnstile middleware is
understood by the tools.Additional options may be provided, such as the control channel,
limits key, and theremote_daemonoptions.  The configuration file
should be compatible with the alternate configuration file described
under theconfigconfiguration option for the Turnstile
middleware.Rate Limit XMLThe XML file used for expressing rate limit configuration is
relatively straightforward, or at least as straightforward as XML can
be.  The top-level element is<limits>; this should contain a
sequence of<limit>elements, each containing a number of<attr>elements.  The specific attributes available for any given
limit class depend on the exact class, but that information is
documented in theattrsattribute of the limit class.  (This
information is suitable for introspection.)The<limit>element has one XML attribute which must be specified:
theclassattribute, which must be set to a “module:class” string
identifying the desired limit class.  The<attr>element also has
a single XML attribute which must be set:name, which identifies
the name of the Limit attribute.  The contents of the<attr>element identify the value for the named attribute.Some limit attributes are lists; for these attributes, the<attr>element must contain one or more<value>elements, whose contents
identify a single item in the attribute list.  Other limit attributes
are dictionaries; for these attributes, again the<attr>element
must contain one or more<value>elements, but now those<value>elements must have the XML attributekeyset to the
dictionary key corresponding to that value.As an example, consider the following limits configuration:<?xml version='1.0' encoding='UTF-8'?>
<limits>
  <limit class="turnstile.limits:Limit">
    <attr name="requirements">
      <value key="pageid">[0-9]+</value>
    </attr>
    <attr name="unit">second</attr>
    <attr name="uri">/page/{pageid}</attr>
    <attr name="value">10</attr>
    <attr name="verbs">
      <value>GET</value>
    </attr>
  </limit>
</limits>In this example, GET access to/page/{pageid}is rate-limited to
10 per second.  Therequirementsattribute may be used to specify
regular expressions to tune the matching of URI components; in this
case, the{pageid}value must be composed of 1 or more digits.
The limit class used is the basicturnstile.limits:Limitlimit
class.Custom Limit ClassesAll limit classes must descend fromturnstile.limits:Limit.  This
admittedly un-Pythonic requirement has a number of advantages,
including the specific machinery which allows limits to be stored into
the Redis database.  Most limit classes only need to worry about theattrsclass attribute and thefilter()method, although theroute()andformat()methods may also be hooked.  For more
information about these methods, see the docstrings provided for their
default implementations inturnstile.limits:Limit.Accessing the Turnstile ConfigurationThe Turnstile configuration is available to preprocessors and to the
Limit classes.  For preprocessors, it is available directly from the
middleware object (the first passed parameter) via theconfigattribute.  (The database handle is also available via thedbattribute, should access to the database be required.)  For thefilter()method of the Limit classes, the configuration is
available in the request environment under theturnstile.confkey.The Turnstile configuration is represented as aturnstile.config:Configobject.  Configuration keys that do not
contain a “.” are available as attributes of this object; for example,
to obtain the configured status value, assuming the Turnstile
configuration is available in theconfvariable, the correct code
would be:status = conf.statusFor those configuration keys which do contain a “.”, the part of the
name to the left of the first “.” becomes a dictionary key, and the
remainder of the name will be a second key.  For example, to access
the value of theredis.connection_pool.connection_classvariable,
the correct code would be:connection_class = config['redis']['connection_pool.connection_class']All values in the configuration are stored as strings.  Configuration
values do not need to be pre-declared in any way; Turnstile ignores
(but maintains) configuration values that it does not use, making
these values available for use by preprocessors and Limit subclasses.For convenience, theturnstile.config:Configclass offers a static
methodto_bool()which can convert a string value to a boolean
value.  The strings “t”, “true”, “on”, “y”, and “yes” are all
recognized as a booleanTruevalue, as are numeric strings which
evaluate to non-zero values.  The strings “f”, “false”, “off”, “n”,
and “no” are all recognized as a booleanFalsevalue, as are
numeric strings which evaluate to zero values.  Any other string value
will causeto_bool()to raise aValueError, unless thedo_raiseargument is given asFalse, in which caseto_bool()will return a booleanFalsevalue.