Package: contexture
> **contexture** _|kənˈteksˌCHər|_>> noun> * the fact or manner of being woven or linked together to form a connected whole.> * a mass of things interwoven together; a fabric.> * the putting together of words and sentences in connected composition; the construction of a text.> * a connected literary structure; a continuous text.Contexture is a thin framework for sharing objects' current state across network. This is useful for distributed structured logging, monitoring, data propagation/collection, synchronization and general elevated states of awareness.Contexture is built on a messaging system (powered by RabbitMQ, see the tail of this document). It solves problems associated with log parsing and directly coupled systems.<!---### Why message queues? I can put my object directly into Cassandra/Senty/whatever.You can, but it tightly couples your system to a given backend. This will make you sad in several ways:1. Flow control. A slow/downed backend might block your code or lose your messages. At some point you will want to restart just one piece of your pipeline.1. Modularity: switching to a different consumer likely requires a change to the source.1. Sharing: adding another consumer also requires a change to the source1. Serialization:--># DesignContexture is designed for use in high availability systems. As such, it has the following features:* low and bounded impact on the running system* low and bounded propagation delay (a few seconds, depending on configuration)* high capacity (depending on configuration, as memory allows)* resistance to problems upstream, such as broker outagesIn other words, your change may not propagate instantly but always within a few seconds, and the channel is wide.The default behavior is to prefer losing messages over blocking or running out of memory(possible depending on messaging rates and queue size limits), this is configurable.# InputObjects go in, objects go out. We can explain that.There are several ways to get your data into the system. One is to think in terms of contexts. A typical context is a collection of objects associated with a request/event, that you wish to monitor.Consider a common pattern:```pythondef handle(request):# Request is receivedurl = request['url']start = time.time()# Stuff happenslog.debug('Starting to fetch %s' % url)result = requests.get(url)log.debug('Status code: %s' % result.status_code)status = result.status_codeif status == 200:log.debug('Content: %s' % result.content)# And we're donelog.debug('Finished in %f seconds' % (time.time() - start))return result...request = {'url': 'http://foo.com'}result = handle(request)```Use Contexture to hold your objects and the above code becomes:```pythonrequest = {'url': 'http://foo.com'}with Context(context=request) as ctx:result = requests.get(request['url'])ctx.status_code = result.status_codeif ctx.status_code == 200:ctx.content = result.content```Done. This produces the following messages:1. object is born, object has one field `url` set to 'http://foo.com'1. object has a field `status_code` set to XXX1. object has a field `content` set to YYY1. object is gone after N secondsThese messages are automatically broadcast to the message bus and, optionally, written to the local logfile. Downstream, you can reconstruct the object or examine its evolution, in realtime.## Usage & integrationContexture is designed to be dropped in with minimal impact on the existing ecosystem.Contexture has several personalities, one of which is `contextmanager`. In most cases it will work without the `with` — the object will properly terminate once it goes out of scope, unless you hold on to a reference. A `with` block is recommended, but you can also `del` the object manually (plus sometimes the object lives beyond the `with` block).> #### Names and keys> Once messages are published to the server, they can be selectively routed by consumers using _routing keys_ and _headers_. By default, all messages aggregate into a single stream, routing lets you pick out just your messages.>> A routing key is an arbitrary dot-separated alphanumeric string, every message sent by Contexture gets one. Contexture tries to smart about generating default keys by deriving them from current module names and stack traces. When unsure, use `name` or `routing_key` argument when constructing a Contexture instance.>> You can see the default name that LC picked by casting it to `str`.>### DirectCreate a new object whenever you need a context.```pythondef handle(request):                    # Outgoing messages (approximately):ctx = Context()                     # {status: 'born', obj: {}}ctx.request = request               # {obj: {request: <request>}}...result = process_request(request)ctx.result = result                 # {obj: {result: <result>}}return                              # {status: 'finished', elapsed: 0.1234}```You may also preload an existing context dict by passing a `context` argument to the initializer.```pythonContext(context={'a': 1, 'foo': 'bar'})  # {status: 'born', obj: {a: 1, foo: 'bar'}}```#### Reserved keywords and direct accessBecause Contexture is not a real dict but a thing of magic, some attribute names cannot be used for direct access (*_*, *context*, *log*, *update*). For example, you cannot do```pythonctx.log = 'something'print ctx.log```Do it by accessing the context dict directly:```pythonctx.update(log='something')print ctx.context['log']```Note the use of `ctx.update()` instead of `ctx.context.update()`. Don't worry about accidentally using a reserved keyword, Contexture will not let you.### SubclassYou can sublass Context. It will try to derive a name/routing_key from the name of your new class.### Object proxyYou have an existing object that you want to monitor and do not want to reimplement it as a subclass of LC. No problem, wrap it:```pythonclass MyObj(object):x = 1@propertydef y(self):return 2def f(self):return 3myobject = MyObj()original_object = myobjectmyobject = Context(obj=myobject)myobject.f() == 3                       # Truemyobject.y == 2                         # Truemyobject.x == 1                         # Truemyobject.y = 10                         # works! y is now part of context, shadowing the originalmyobject.y == 10                        # Trueoriginal_object.y == 2                  # Truemyobject.x = 111myobject.x == original_object.x == 1    # True, as expected```This wrapped object behaves like the original object (function calls, @properties, etc.), but it is also a context: it will capture all attribute assignments and allow creation of new attributes (shadowing the existing read-only ones).In other words, wrap an existing object in Contexture and it will do its best to make sure neither the object nor the code that is using it notice a difference.<!--- rewrite maybe? -->### ExclusionsYou have a complex dict but you only care to broadcast a part of it. Add the private bits to `ignore` and it will not be reported.```pythonmydict = dict(x=1, y=2, privatestuff=whatever)ctx = Context(context=mydict,ignore=['privatestuff', 'foo'])   # obj: {x: 1, y: 2}ctx.x = 5                                       # obj: {x: 5}ctx.foo = 1234                                  # nothing```### Actually logging to the message busIt slices, it dices, it logs! You can still use it as a conventional logger, via the `log` attribute.```pythonctx.log.info('OHAI')```Use the context to format your messages```pythonctx = Context(context={'foo': 1})                # obj: {foo: 1}ctx.bar = 2                                         # obj: {bar: 2}...ctx.log.debug('my foo is {foo} and my bar is {bar}')    # obj:{}, message: 'my foo is 1 and my bar is 2'```### Logging to a logfileProvide your own logger instance and Contexture will log context changes to a log file for you, automatically:```pythonctx = Context(logger=logging.getLogger(__name__))    # <name> <id>: status = bornctx.update(x=1, y=2)                                 # <name> <id>: x = 1, y = 2```Note that this will work even without an AMQP handler configured.## Usage notesData must be JSON serializeable. The entire context entity must fit into JSON.### Can't we give an attribute a serializer?### pika.exceptions.AMQPConnectionErrorThis means the backend could not connect to the broker. This harms nothing, only means your messages won't make it upstream. See below.### Will this break my code if X happens downstream?No.Inside the handler, your messages are buffered in a queue and picked up by a publisher thread every second. If the publisher cannot publish to the RabbitMQ server, it will stop publishing, the queue will fill up and the new messages will be discarded. The backend will keep trying to reconnect and things will return to normal once it does.This means Contexture can survive some broker downtime with no loss of traffic. How long — depends on your message rates and the size of the queue (currently 5000 but we can make this configurable).### Clean exitIt is possible to ensure the context transmits all of its messages before the process exits. Declare your context with ``wait=True`` and it will block on exit until the publish queue is empty.The old way was to call `time.sleep(2)` before exiting your program, that still works.### Unicode headersPika does not like your unicode headers. If using `unicode_literals`, be sure to do something like:```pythonContext(headers={b'myheader': b'something'})```### Packing & sequencingWhen updating multiple fields, you have a choice of how the updates will be packedon the messages stream. For example, this will create 3 separate messages:ctx.foo = 1ctx.bar = 2ctx.baz = 5and this only one:ctx.update(foo=1, bar=2, baz=5)The spreading of the updates exposes more of the internal behavior and invidivual timing.When not needed, it is preferable to pack multiple udpates into one.Same goes for preloading contexts during initialization.### Transient objectsNormally the context object announces its birth and death. You can bypass that and send your object upstream in the most minimalist way:```pythondef push(obj):'Convenience function for publishing a short-lived object'Context(name='my.routing.key', transient=True, context=obj)...myobject = dict(foo='whatever', ...)...push(myobject)```### Long lived objectsThe other extreme. You can think of a long lived object as a simple streaming database handle. That is, you ignore the object's own lifetime and treat it as a delta emitter, while listening for those deltas on the other end.```pythondb = Context(name='faux_db')db.mystatus = 'now this'...db.mystatus = 'and now that'```And on the other end you might do (see the section on _lcmon_ below).$ lcmon -v -r faux_db -k obj{"mystatus": "now this"}{"mystatus": "and now that"}### Objects with custom IDs```pythonctx = Context(guid='1234')```Context objects are assigned UUIDs automatically, but you can provide your own. This may be helpful with transient objects in particular.## ConfigurationPut this in your config:```[logger_contexture]level=DEBUGhandlers=AMQPHandlerqualname=contexturepropagate=0[handler_AMQPHandler]class=contexture.backend.amqp_handler.AMQPHandlerlevel=DEBUGargs=("amqp://guest:guest@localhost:5672/%2F", "lc-topic", "topic")```Add the logger to loggers and handler to handlers (see `contexture/config.conf` for an example). The arguments to AMQP handler are:1. AMQP connection string1. Exchange to which to publish (will be created as needed)1. Exchange type1. (optional) A dict of headers to be included with every message# OutputSo you know everything (almost) about getting data onto the message bus. Now what?Inspect the messages as they fly by or grab them from the stream for your own devilish purpose.## Monitoring: lcmonIncluded in _contexture_ package is a handy monitoring utility.$ lcmon -husage: lcmon [-h] [-r RKEYS] [-a ARGS [ARGS ...]] [-x {all,any}] [-e EXCHANGE][-H HOSTNAME] [-u URL] [-q QUEUE] [-s] [-k KEYS [KEYS ...]] [-c][-p] [-v]Simple AMQP monitoroptional arguments:-h, --help            show this help message and exit-H HOSTNAME, --hostname HOSTNAMEAMQP server hostname (default: localhost)-u URL, --url URL     fully qualified url (overrides hostname) (default:None)-e EXCHANGE, --exchange EXCHANGEexchange to bind to (default: lc-topic)-q QUEUE, --queue QUEUEconsume from an existing queue (default: None)-r RKEY [RKEY ...], --rkey RKEY [RKEY ...]routing keys (default: ['#'])-a ARG [ARG ...], --arg ARG [ARG ...]binding arguments (key=value pairs) (default: None)-x {all,any}, --xmatch {all,any}x-match (default: all)-s, --stdin           get input from stdin instead of queue (default: False)-k KEYS [KEYS ...], --keys KEYS [KEYS ...]keys to extract (*key for all matches) (default: None)-c, --collate         extract collated objects from message stream (default:None)-p, --pretty          pretty print (default: False)-v, --verboseThis will print traffic on the message bus in realtime, until terminated.### -H, --hostnameBy default _lcmon_ connects to localhost. Open an SSH tunnel (port 5672) or provide the hosname explicitly:$ lcmon -H <amqp server host>This will expand to a default URL of `amqp://guest:guest@<hostname>:5672/%2F`. Alternatively you can use `-U` to provide the full URL.### -q, --queueConsume from an existing queue.By default, _lcmon_ creates an anonymous queue which disappears after the connection is closed (this happens _almost_ always). You can declare a more permanent queue and have messages survive _lcmon_ and even broker restarts.**Warning!** Take care to not consume from queues not belonging to you. This is a destructive operation.### -r, --rkeysOne or more routing keys or wildcards to filter on. Wildcards will only work with topic exchanges.### Binding args (-a)Arguments for use with headers exchange. Use with `-x`.### -s, --stdinYou can store the output in a file, then read it back in and reprocess with options `-c`, `-k`, `-v` and `-p`. This is useful for temporary local storage (`-v` is important here):$ lcmon -v > myfile^C$ lcmon -sc < myfile### -k, --keysExtract occurences of keys from the object.Objects can be big, and you might be interested only in a subset of keys. Here is a real message:{"status": "finished", "obj": {"elapsed": 2.7338700294494629}, "id": "c529c182-62b8-49db-a0aa-2ba4cc991af2", "handler_id": "76bb5a4e-34b7-43a0-8ffc-19e448d33ea0", "elapsed": 1.1670801639556885, "queue": 0, "time_in": 1360720859.8556099, "time_out": 1360720861.0226901}Suppose you only care about elapsed times. Running `lcmon -k elapsed` prints this:{'elapsed': 1.1670801639556885}Notice there are actually two `elapsed` fields in the message. By default, `-k` prints the first one found by breadth-first search. Prefixing the name of the key with `*` gets all of them:In [6]: monitor.extract_keys(obj, ['elapsed'])Out[6]: {'elapsed': 1.1670801639556885}In [7]: monitor.extract_keys(obj, ['*elapsed'])Out[7]: {'elapsed': (1.1670801639556885, 2.733870029449463)}### -c, --collateReconstruct complete context objects from the stream, so instead of a sequence likestatus: 'born', obj: {}obj: {x: 1, y: 2}obj: {x: 5}status: 'finished'you will see the object{x: 5, y: 2}This only works if the object terminates properly, and is not useful with long-lived objects. It does work with transient objects.Note `-k` option works on collated objects as well.### -vInclude the headers and routing key in the final object, resulting in a slightly different structure. You will want this often (hint: grep).## Consuming: messages and objectsThe easiest part.```pythonfrom contexture.monitor import messages, objectsfor message in monitor.messages(binding_keys=['#']):outer_obj = message['object']my_obj = outer_obj['obj']my_id = outer_obj['id']```The heart of _lcmon_, `monitor` module provides two important functions: `messages` and `objects`. They return iterators over messages and collated objects. The iterators are also _contextmanagers_, so you can do:```pythonwith messages() as m_iter:for message in m_iter:dostuff()```They take same arguments as `lcmon`, see `monitor.py` for details. Again, when using a named queue, remember to delete it when done (`message.channel.queue_delete(<name>)` works, see below).A full example with objects. This also demonstrates publishing:```pythonobjects = monitor.objects(verbose=True, capture_messages=True, queue='analytics.es')channel = objects.channelprint 'Pushing objects to ElasticSearch River'buffer_.clear()for obj in objects:buffer_.append({'index': dict(_ttl='7d',_index='lc',_type=obj.pop('rkey'),_id=obj.pop('id'),)})buffer_.append(obj)channel.basic_publish('elasticsearch','elasticsearch','\n'.join(map(json.dumps, buffer_)) + '\n')```### Queue safety* Q: What if I don't consume my messages fast enough?* A: You laggard!But don't worry, RabbitMQ has tolerance for the likes of you. By default, Contexture declares all queues with a TTL (usually 60 seconds) — messages left unconsumed longer than that are automatically sent to /dev/null, providing a bound on queue growth. So even if your consumer gets terminally stuck, the world will go on.# StorageLeft as an exercise.# MiscellaneaWhat's in the box:* an ActiveRecord-style data container (`Context`), for exposing your crazy world to the backend* an AMQP backend handler, for marshalling updates to the message bus* utilities for monitoring the message bus (lcmon)Contexture helps you remotely monitor and analyze your code, in realtime, as it runs on one or more machines across a network. More backends can be added with relative ease.Design considerations include:* Easy integration: minimal effort is required to start sending and receiving context objects.* Zero impact: LC is guaranteed to not impact the rest of the system. It isasyncronous and handles downstream failures gracefully. Worstthat it will do is a little memory usage (= size of the internal queue) and loss of updates(once the queue fills up).### RabbitMQThe default backend is RabbitMQ. Things to know about AMQP:* The server is a __broker__.* __Routing key__ and __headers__ are assigned to the message by the publisher and are used by thebroker to route the message.* __Exchange__ is where the published message arrives first.* The consumer gets its messages from a __queue__. To start receiving messages the consumer mustfirst declare a queue. Queues are often dynamic and disposable, but can be made permanent.* __Binding__ routes messages from exchange to queue. Bindings can use routing keysand/or headers to filter messages.----## What's wrong with direct coupling?1. Producer -> consumer dependency: the consumer must be up and runnning, and be able to consume messages fast enough.1. Consumer -> producer dependency: the producer is responsible for knowing how to talk to the consumer and for maintaining an open channel.1. Complex systems and intercepting streams: adding multiple consumers and/or producers is a challenge, as is reloading individual components in long running processes.These problems can be addressed by implementing plumbing in all of your components or by using a buffered message queue to connect them.## blah* how is this better than syslog?1. updates assemble back into objects2. context objects are linkable## CongratulationsYou have read the whole thing.