crits/crits — pull request #102 by wxsBSD (2014-06-13 16:33:33+00:00)
Title: Distributed services

Add support for distributed services. This is a pretty intrusive change and is going to require significant testing. I will be putting up a pull request in crits_services which has the other half of these changes.

As an example I have made the yara service optionally distributed. There are now some extra options to the yara service if you want to experiment. They are the "distribution_url" which you should point to the full AMQP url you want to use (I'm using 'amqp://guest:guest@amqp.local//' - and yes, the double slash is important). If you choose not to distributed yara you can just leave that blank and there will be no changes.

The other new options are an exchange (defaults to 'woodchipper') and routing key (defaults to 'file.crits'). This is needed for distribution. The way we have our queues set up there is one exchange into which everything should be sent (woodchipper), and multiple queues bound to that one exchange. I currently have a queue named 'yara' with a routing key of 'file.*'. When a message is sent from the yara service to the queue it will use the routing key 'file.crits', which will ensure it is routed into the 'yara' queue.

I have one other queue bound to the 'woodchipper' exchange. This queue is called 'peinfo' and uses a routing key of 'file.crits.peinfo'. The nice thing about these particular routing keys is that if we have the yara scanner listening to the queue we can make it work nicely with files that are not from CRITs. If you have a file you would like to scan that does not come from CRITs you just need to send it with the routing key 'file.' and the appropriate message format and it will be picked up by the yara scanner. This allows us to have one yara scanner for CRITs and anything else simultaneously. Other services that want to be distributed can use the appropriate routing key ('file.crits.peinfo') and the file will only go to the peinfo queue. All of this is configurable and can be customized in the service config options.

The other piece to the puzzle is something which reads the message off the queue, pulls the object out of CRITs via the API, scans it and puts the results back on the queue via the API. I have a working prototype of this but need to work with Murad a bit more to clean it up before I feel comfortable sharing it. For now, if you want to test I'd start with just ensuring messages are landing on the queue properly.

In doing this work I've also changed services by removing the notion of a "context". These existed from a time before we were using mongoengine and it made passing data into the service sane, but now with mongoengine we can just pass the object into the service. It makes the code much cleaner, but does require that all services be modified to handle an object. I've done it in crits_services for all the services there, but any custom services not in that tree will need to be modified.

I've also removed the notion of LocalAnalysisSource and LocalAnalysisDestination. They are never used anywhere and were just complicating the code. If someone is using these we can see about adding them back in, but given the neglect they have received I'm not sure anyone is actually using them.