Package: whip
OverviewWhip, the who, what, where and when about IP address data.Whip provides a fast IP information lookup service that scales to large data
sets. Whip can efficiently answer queries for a specific IP address, optionally
refined by a timestamp.Feature summary:Fast lookups for IP addresses, optionally limited to a specific timestamp.
This means that data that (slowly) changes over time can be queried at any
known point in history.Transparent support for both IPv4 and IPv6 addresses and rangesSupport for incremental loading. This means new data can be added to an
existing database. This is useful to ingest weekly data set snapshots.Efficient storage format (data stored as ranges, history as diffs)Transparent support for gzip compressed input data the file name ends with.gzWhip can handle almost any data set containing information about IP address
ranges. Whip does not limit the type of data that is associated to an IP address
or IP address range, so it can be used for a variety of data sets, for instance
IP geolocation data sets. Note that Whip itself doesnotcome with any data
sets.A source file should be seen as a ‘snapshot’ of an evolving data set at a
specific point in time. During the loading phase, Whip combines all snapshots
(with different timestamps) and constructs the history of all records in a way
that can be queried efficiently. It does so by building an index of IP ranges
and associating data to each range (e.g. geolocation tags), and keeping history
for each distinct range.InstallationUse a virtualenv to install Whip and its dependencies. To install from a source
tree:$ pip install -r requirements.txtThese are the current dependencies:Python3.3+ (no Python 2 support!)Plyvelto accessLevelDBfrom PythonFlaskfor the REST APIAaarghfor the command line toolUltraJSON (ujson)for fast JSON encoding and decodingMsgpackfor Msgpack encoding and decodingUsageCommand line interfaceMost functionality is provided by thewhip-clicommand line tool. Detailed
usage information:$ whip-cli --helpTo load input data intomy.db(creating it when necessary):$ whip-cli --db my.db load input-file-1.json.gz input-file-2.json.gzTo serve the database over a REST API:$ whip-cli --db my.db serveThe REST API can also be deployed using WSGI e.g. using gunicorn/nginx.REST APIThe REST API supports these queries:To retrieve the latest record for an IP address:GET /ip/1.2.3.4To obtain a specific record for an IP address:GET /ip/1.2.3.4?datetime=2013-05-15To obtain the complete history for an IP address:GET /ip/1.2.3.4?datetime=allIn each case, the response will be anapplication/jsonencoded document,
even if no hit was found, in which case the result will be an empty JSON
document. HTTP status codes are only used to signify errors.Input data formatThe source file format is simple: it’s just a a text file with one
JSON-formatted document on each line. Each document can contain arbitrary
information about a range of IP addresses.Whip itself requires three fields:begin: begin of IP address range (inclusive), e.g.1.0.0.0end: end of IP address range (inclusive), e.g.1.0.255.255datetime: time stamp for this record, e.g.2013-02-28In addition to these fields, each document may contain arbitrary key/value
pairs, e.g."country".An input data file must follow these rules:Each range in the file must span at least 1 IP address, but can span an
arbitrary number of consecutive IP addresses (not limited by net block/CIDR),
specified in thebeginandendfields.Ranges in a single input file must not overlap, but there may be gaps between
ranges (in case no information is available for that range).Ranges in a single input file must be sorted by IP address. IPv4 ranges must
sortbeforeIPv6 ranges, since Whip uses RFC3493 IPv4-mapped IPv6 addresses
internally.Timestamps, specified indatetimefield, should be in ISO8601 format,
since Whip depends on the lexicographic ordering of the input strings.All timestamps in a single input file must be the same. Yes, this adds
redundancy, but avoids the need for header lines or out-of-band metadata.WarningIf the input data does not follow the above rules, bad things may happen,
including database corruption.An example input document looks like this (formatted on multiple lines for
clarity):{
    "begin": "1.0.0.0",
    "end": "1.255.255.255",
    "datetime": "2013-02-28",
    "location": "Amsterdam",
    "some-other-data": "anything-you-like"
}A single input file with many of these documents looks like this:{"begin": "1.0.0.0", "end": "1.255.255.255", "datetime": "2013-02-28", "...": "..."}
{"begin": "2.0.0.0", "end": "2.255.255.255", "datetime": "2013-02-28", "...": "..."}
{"begin": "11.0.0.0", "end": "11.0.255.255", "datetime": "2013-02-28", "...": "..."}Whip can load many of these input files (e.g. weekly snapshots for a longer
period of time) in a single loading pass.Ideas / TODOPerform range scan on in-memory structure instead seeking on a DB iterator.
This means Whip must load all keys in memory on startup (in anarray.array);
usebisect.bisect_rightto find the right entry, then simplydb.get()for
the actual value. To avoid scanning the complete database on startup, the
keyspace should be split using a key prefix: one part keeps both the keys and
values (full database), the other part only keeps the keys. The latter will be
scanned and loaded into memory upon startup. For ~25000000 IPv4 addresses,
keeping the index in memory only requires 100MB of RAM, and lookups would only
issuedb.get()for keys that are known to exist.Update: experiments on a big database containing most IP ranges in use show
this isnotany faster than doing the actual seek, sinceit.seek()takes
just as long asdb.get(). This means a lot of memory will be used to improve
performance for non-hits (in which case no DB calls are made).Try out LMDB instead of LevelDBPluggable storage backends (e.g. HBase)